You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2020/03/16 06:42:43 UTC

[GitHub] [incubator-tvm] icemelon9 opened a new pull request #5078: [DOC] Add doc for Relay op strategy

icemelon9 opened a new pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078
 
 
   Thanks for contributing to TVM!   Please refer to guideline https://docs.tvm.ai/contribute/ for useful information and tips. After the pull request is submitted, please request code reviews from [Reviewers](https://github.com/apache/incubator-tvm/blob/master/CONTRIBUTORS.md#reviewers) by @ them in the pull request thread.
   
   cc @tqchen @kevinthesun @comaniac @tmoreau89 @yzhliu 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393454565
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393245677

##########
File path: docs/dev/relay_op_strategy.rst
##########
@@ -0,0 +1,256 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``). The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+ def add_implementation(self, compute, schedule, name="default", plevel=10)

Review comment:
I agree with sticking to the keywords here

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [incubator-tvm] comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393159935
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
 
 Review comment:
   - Remove the comma in the beginning.
   - Replace the comma with a period at the end. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393250506
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
+Similarly, we need to first define the ``FTVMStrategy`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @override_native_generic_func("my_new_op_strategy")
+    def my_new_op_strategy(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.my_new_op),
+            wrap_topi_schedule(topi.generic.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @my_new_op_strategy.register("cpu")
+    def my_new_op_strategy_cpu(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.x86.my_new_op),
+            wrap_topi_schedule(topi.x86.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+In this example, we use two wrapper function that wrap the topi compute and
+schedule function to conform with the required function signature. Usually we
+need to write a customized compute wrap function to retrieve different fields
+from op attributes. After that, we can then register this strategy to the new
+operator by
+
+.. code:: python
+
+    register_strategy("my_new_op", strategy.my_new_op_strategy)
+
+
+Advanced strategy function
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The example above only shows the very basic strategy function.
+In this part, we will show a few advanced ways to define op strategy.
+
+First, we can add multiple implementations that use different algorithms to the
+same operator:
+
+.. code:: python
+
+    strategy.add_implementation(
+        wrap_compute_my_op(my_op_compute1),
+        wrap_topi_schedule(my_op_schedule1),
+        name="my_implementation1",
+        plevel=10)
+
+    if some_condition:
+        strategy.add_implementation(
+            wrap_compute_my_op(my_op_compute2),
+            wrap_topi_schedule(my_op_schedule2),
+            name="my_implementation2",
+            plevel=15)
+
+In this example, we add two implementations to the op strategy where
+implementation 2 is added with certain condition. ``my_implementation2`` will be
+used to compile this operator when ``some_condition`` is true as it has higher
+priority level (this could be changed if certain implementation is an AutoTVM
+template. See `Select implementation from op strategy`_ for more
+details). Otherwise, ``my_implementation1`` is used.
+
+We can extend the example above to third party library implementation. For
+example, we can add the implementation that invokes kernel in the third party
+library when the library is included in the target.
+
+.. code:: python
+
+    if "some_lib" in target.libs:
+        strategy.add_implementation(
+            wrap_compute_my_op(my_op_compute_lib),
+            wrap_topi_schedule(my_op_schedule_lib),
+            name="my_implementation_lib",
+            plevel=20)
+
+
+Further, we can add implementation specialized for a certain range of shapes.
+The code below shows an example of dense strategy that adds an implementation
+that is specialized for ``m`` greater than 16. The main difference between
+hardcode python condition like examples above and specialized condition is that
+it allows TVM to generate multiple kernels when the input tensors have symbolic
+shapes. The compile engine will generate a dispatch function that invokes the
+specialized kernel first when the corresponding condition is met; otherwise,
 
 Review comment:
   remove "first" to improve flow

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] tmoreau89 merged pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

tmoreau89 merged pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r394634872

Review comment:
Ack, I found the examples indeed; that was the result of writing feedback in real time and not removing the feedback after finishing my pass

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on issue #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on issue #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#issuecomment-599906691
 
 
   @comaniac I replaced "my_new_op" with some real ops now. Could you take a look again?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393160975
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
 
 Review comment:
   Remove "But".

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393158245

Review comment:
The API here seems a bit confuse. Specifically, it doesn't use any keywords mentioned in the above paragraph. My feeling is this section focuses on components on C++ backend, so would that be better to move this API intro to the next section along with other Python interfaces?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [incubator-tvm] comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393162269
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
+Similarly, we need to first define the ``FTVMStrategy`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @override_native_generic_func("my_new_op_strategy")
+    def my_new_op_strategy(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.my_new_op),
+            wrap_topi_schedule(topi.generic.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @my_new_op_strategy.register("cpu")
+    def my_new_op_strategy_cpu(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.x86.my_new_op),
+            wrap_topi_schedule(topi.x86.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
 
 Review comment:
   Maybe we can move the introduction to `add_implementation` here?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393465956

Review comment:
done

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393473787
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
+Similarly, we need to first define the ``FTVMStrategy`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @override_native_generic_func("my_new_op_strategy")
+    def my_new_op_strategy(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.my_new_op),
+            wrap_topi_schedule(topi.generic.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @my_new_op_strategy.register("cpu")
+    def my_new_op_strategy_cpu(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.x86.my_new_op),
+            wrap_topi_schedule(topi.x86.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+In this example, we use two wrapper function that wrap the topi compute and
+schedule function to conform with the required function signature. Usually we
+need to write a customized compute wrap function to retrieve different fields
+from op attributes. After that, we can then register this strategy to the new
+operator by
+
+.. code:: python
+
+    register_strategy("my_new_op", strategy.my_new_op_strategy)
+
+
+Advanced strategy function
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The example above only shows the very basic strategy function.
+In this part, we will show a few advanced ways to define op strategy.
+
+First, we can add multiple implementations that use different algorithms to the
+same operator:
+
+.. code:: python
+
+    strategy.add_implementation(
+        wrap_compute_my_op(my_op_compute1),
+        wrap_topi_schedule(my_op_schedule1),
+        name="my_implementation1",
+        plevel=10)
+
+    if some_condition:
+        strategy.add_implementation(
+            wrap_compute_my_op(my_op_compute2),
+            wrap_topi_schedule(my_op_schedule2),
+            name="my_implementation2",
+            plevel=15)
+
+In this example, we add two implementations to the op strategy where
+implementation 2 is added with certain condition. ``my_implementation2`` will be
+used to compile this operator when ``some_condition`` is true as it has higher
+priority level (this could be changed if certain implementation is an AutoTVM
+template. See `Select implementation from op strategy`_ for more
+details). Otherwise, ``my_implementation1`` is used.
+
+We can extend the example above to third party library implementation. For
+example, we can add the implementation that invokes kernel in the third party
+library when the library is included in the target.
+
+.. code:: python
+
+    if "some_lib" in target.libs:
+        strategy.add_implementation(
+            wrap_compute_my_op(my_op_compute_lib),
+            wrap_topi_schedule(my_op_schedule_lib),
+            name="my_implementation_lib",
+            plevel=20)
+
+
+Further, we can add implementation specialized for a certain range of shapes.
+The code below shows an example of dense strategy that adds an implementation
+that is specialized for ``m`` greater than 16. The main difference between
+hardcode python condition like examples above and specialized condition is that
+it allows TVM to generate multiple kernels when the input tensors have symbolic
+shapes. The compile engine will generate a dispatch function that invokes the
+specialized kernel first when the corresponding condition is met; otherwise,
+invoke the kernel that has no associated specialized condition (``dense_common``
+in this example). This part is still work in progress. More details will be
+provided after it is done.
+
+.. code:: python
+
+    def dense_strategy(attrs, inputs, out_type, target):
+        m = inputs[0].shape[0]
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_dense(dense_compute1),
+            wrap_topi_schedule(dense_schedule1),
+            name="dense_common")
+
+        with tvm.te.SpecializedCondition(m > 16):
+            strategy.add_implementation(
+                wrap_compute_dense(dense_compute2),
+                wrap_topi_schedule(dense_schedule2),
+                name="dense_for_large_m",
+                plevel=15)
+
+        return strategy
+
+
+Register strategy for a new target
+----------------------------------
+
+There are two ways to register strategies for a new target. The more
+straightforward one is adding a new target file in the directory
+``python/tvm/relay/op/strategy``. You only need to customize the strategy for
+ops that have been implemented for this new target and reuse the generic
+strategies for the rest.
+
+Alternatively, you can also register the strategy for the new target outside the
+TVM python library. The following code snippet shows an example how to do
+so. You can find more examples in ``vta/python/vta/top/op.py``.
+
+.. code:: python
+
+    @relay.op.strategy.someop_strategy.register("mytarget")
+    def someop_strategy_mytarget(attrs, inputs, out_type, target):
+        ...
+
+
+Select implementation from op strategy
+--------------------------------------
+
+During the compilation, Relay compile engine needs to determine which
+implementation to use for an operator when there are multiple. The selection
+policy works as follows.
+
+When the input tensors to an operator or a fused op all have constant shapes,
+the compile engine first finds the best implementation based on AutoTVM tuning
+logs. If there is no implementation that is an AutoTVM template or all AutoTVM
+templates have fallback configs, the implementation with highest priority level
+will then be chosen. Implementations with same priority level in this case leads
+to an undefined behavior, and any of them might be selected.
 
 Review comment:
   Agree with @comaniac. It might be too strict to trigger an error just because two implementation has the same plevel.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393473477

Review comment:
The example is provided in the later part. Specialized condition currently is only used to deal with symbolic shapes. Either data layout or data type matching should be done in python. I added some more explanation here. Do you think it's clear enough?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393454315
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
+Similarly, we need to first define the ``FTVMStrategy`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @override_native_generic_func("my_new_op_strategy")
+    def my_new_op_strategy(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.my_new_op),
+            wrap_topi_schedule(topi.generic.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @my_new_op_strategy.register("cpu")
+    def my_new_op_strategy_cpu(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.x86.my_new_op),
+            wrap_topi_schedule(topi.x86.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+In this example, we use two wrapper function that wrap the topi compute and
+schedule function to conform with the required function signature. Usually we
+need to write a customized compute wrap function to retrieve different fields
+from op attributes. After that, we can then register this strategy to the new
+operator by
+
+.. code:: python
+
+    register_strategy("my_new_op", strategy.my_new_op_strategy)
+
+
+Advanced strategy function
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The example above only shows the very basic strategy function.
+In this part, we will show a few advanced ways to define op strategy.
+
+First, we can add multiple implementations that use different algorithms to the
+same operator:
+
+.. code:: python
+
+    strategy.add_implementation(
+        wrap_compute_my_op(my_op_compute1),
+        wrap_topi_schedule(my_op_schedule1),
+        name="my_implementation1",
+        plevel=10)
+
+    if some_condition:
+        strategy.add_implementation(
+            wrap_compute_my_op(my_op_compute2),
+            wrap_topi_schedule(my_op_schedule2),
+            name="my_implementation2",
+            plevel=15)
+
+In this example, we add two implementations to the op strategy where
+implementation 2 is added with certain condition. ``my_implementation2`` will be
+used to compile this operator when ``some_condition`` is true as it has higher
+priority level (this could be changed if certain implementation is an AutoTVM
+template. See `Select implementation from op strategy`_ for more
+details). Otherwise, ``my_implementation1`` is used.
+
+We can extend the example above to third party library implementation. For
+example, we can add the implementation that invokes kernel in the third party
+library when the library is included in the target.
+
+.. code:: python
+
+    if "some_lib" in target.libs:
+        strategy.add_implementation(
+            wrap_compute_my_op(my_op_compute_lib),
+            wrap_topi_schedule(my_op_schedule_lib),
+            name="my_implementation_lib",
+            plevel=20)
+
+
+Further, we can add implementation specialized for a certain range of shapes.
+The code below shows an example of dense strategy that adds an implementation
+that is specialized for ``m`` greater than 16. The main difference between
+hardcode python condition like examples above and specialized condition is that
+it allows TVM to generate multiple kernels when the input tensors have symbolic
+shapes. The compile engine will generate a dispatch function that invokes the
+specialized kernel first when the corresponding condition is met; otherwise,
+invoke the kernel that has no associated specialized condition (``dense_common``
+in this example). This part is still work in progress. More details will be
+provided after it is done.
+
+.. code:: python
+
+    def dense_strategy(attrs, inputs, out_type, target):
+        m = inputs[0].shape[0]
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_dense(dense_compute1),
+            wrap_topi_schedule(dense_schedule1),
+            name="dense_common")
+
+        with tvm.te.SpecializedCondition(m > 16):
+            strategy.add_implementation(
+                wrap_compute_dense(dense_compute2),
+                wrap_topi_schedule(dense_schedule2),
+                name="dense_for_large_m",
+                plevel=15)
+
+        return strategy
+
+
+Register strategy for a new target
+----------------------------------
+
+There are two ways to register strategies for a new target. The more
+straightforward one is adding a new target file in the directory
+``python/tvm/relay/op/strategy``. You only need to customize the strategy for
+ops that have been implemented for this new target and reuse the generic
+strategies for the rest.
+
+Alternatively, you can also register the strategy for the new target outside the
+TVM python library. The following code snippet shows an example how to do
+so. You can find more examples in ``vta/python/vta/top/op.py``.
+
+.. code:: python
+
+    @relay.op.strategy.someop_strategy.register("mytarget")
+    def someop_strategy_mytarget(attrs, inputs, out_type, target):
+        ...
+
+
+Select implementation from op strategy
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393161533
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
 
 Review comment:
   s/different implementation/different implementations

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393167447
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
+Similarly, we need to first define the ``FTVMStrategy`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @override_native_generic_func("my_new_op_strategy")
+    def my_new_op_strategy(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.my_new_op),
+            wrap_topi_schedule(topi.generic.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @my_new_op_strategy.register("cpu")
+    def my_new_op_strategy_cpu(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.x86.my_new_op),
+            wrap_topi_schedule(topi.x86.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+In this example, we use two wrapper function that wrap the topi compute and
+schedule function to conform with the required function signature. Usually we
+need to write a customized compute wrap function to retrieve different fields
+from op attributes. After that, we can then register this strategy to the new
+operator by
+
+.. code:: python
+
+    register_strategy("my_new_op", strategy.my_new_op_strategy)
+
+
+Advanced strategy function
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The example above only shows the very basic strategy function.
+In this part, we will show a few advanced ways to define op strategy.
+
+First, we can add multiple implementations that use different algorithms to the
+same operator:
+
+.. code:: python
+
+    strategy.add_implementation(
+        wrap_compute_my_op(my_op_compute1),
+        wrap_topi_schedule(my_op_schedule1),
+        name="my_implementation1",
+        plevel=10)
+
+    if some_condition:
+        strategy.add_implementation(
+            wrap_compute_my_op(my_op_compute2),
+            wrap_topi_schedule(my_op_schedule2),
+            name="my_implementation2",
+            plevel=15)
+
+In this example, we add two implementations to the op strategy where
+implementation 2 is added with certain condition. ``my_implementation2`` will be
+used to compile this operator when ``some_condition`` is true as it has higher
+priority level (this could be changed if certain implementation is an AutoTVM
+template. See `Select implementation from op strategy`_ for more
+details). Otherwise, ``my_implementation1`` is used.
+
+We can extend the example above to third party library implementation. For
+example, we can add the implementation that invokes kernel in the third party
+library when the library is included in the target.
+
+.. code:: python
+
+    if "some_lib" in target.libs:
+        strategy.add_implementation(
+            wrap_compute_my_op(my_op_compute_lib),
+            wrap_topi_schedule(my_op_schedule_lib),
+            name="my_implementation_lib",
+            plevel=20)
+
+
+Further, we can add implementation specialized for a certain range of shapes.
+The code below shows an example of dense strategy that adds an implementation
+that is specialized for ``m`` greater than 16. The main difference between
+hardcode python condition like examples above and specialized condition is that
+it allows TVM to generate multiple kernels when the input tensors have symbolic
+shapes. The compile engine will generate a dispatch function that invokes the
+specialized kernel first when the corresponding condition is met; otherwise,
+invoke the kernel that has no associated specialized condition (``dense_common``
+in this example). This part is still work in progress. More details will be
+provided after it is done.
+
+.. code:: python
+
+    def dense_strategy(attrs, inputs, out_type, target):
+        m = inputs[0].shape[0]
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_dense(dense_compute1),
+            wrap_topi_schedule(dense_schedule1),
+            name="dense_common")
+
+        with tvm.te.SpecializedCondition(m > 16):
+            strategy.add_implementation(
+                wrap_compute_dense(dense_compute2),
+                wrap_topi_schedule(dense_schedule2),
+                name="dense_for_large_m",
+                plevel=15)
+
+        return strategy
+
+
+Register strategy for a new target
 
 Review comment:
   ditto

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393167739
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
+Similarly, we need to first define the ``FTVMStrategy`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @override_native_generic_func("my_new_op_strategy")
+    def my_new_op_strategy(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.my_new_op),
+            wrap_topi_schedule(topi.generic.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @my_new_op_strategy.register("cpu")
+    def my_new_op_strategy_cpu(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.x86.my_new_op),
+            wrap_topi_schedule(topi.x86.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+In this example, we use two wrapper function that wrap the topi compute and
+schedule function to conform with the required function signature. Usually we
+need to write a customized compute wrap function to retrieve different fields
+from op attributes. After that, we can then register this strategy to the new
+operator by
+
+.. code:: python
+
+    register_strategy("my_new_op", strategy.my_new_op_strategy)
+
+
+Advanced strategy function
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The example above only shows the very basic strategy function.
+In this part, we will show a few advanced ways to define op strategy.
+
+First, we can add multiple implementations that use different algorithms to the
+same operator:
+
+.. code:: python
+
+    strategy.add_implementation(
+        wrap_compute_my_op(my_op_compute1),
+        wrap_topi_schedule(my_op_schedule1),
+        name="my_implementation1",
+        plevel=10)
+
+    if some_condition:
+        strategy.add_implementation(
+            wrap_compute_my_op(my_op_compute2),
+            wrap_topi_schedule(my_op_schedule2),
+            name="my_implementation2",
+            plevel=15)
+
+In this example, we add two implementations to the op strategy where
+implementation 2 is added with certain condition. ``my_implementation2`` will be
+used to compile this operator when ``some_condition`` is true as it has higher
+priority level (this could be changed if certain implementation is an AutoTVM
+template. See `Select implementation from op strategy`_ for more
+details). Otherwise, ``my_implementation1`` is used.
+
+We can extend the example above to third party library implementation. For
+example, we can add the implementation that invokes kernel in the third party
+library when the library is included in the target.
+
+.. code:: python
+
+    if "some_lib" in target.libs:
+        strategy.add_implementation(
+            wrap_compute_my_op(my_op_compute_lib),
+            wrap_topi_schedule(my_op_schedule_lib),
+            name="my_implementation_lib",
+            plevel=20)
+
+
+Further, we can add implementation specialized for a certain range of shapes.
+The code below shows an example of dense strategy that adds an implementation
+that is specialized for ``m`` greater than 16. The main difference between
+hardcode python condition like examples above and specialized condition is that
+it allows TVM to generate multiple kernels when the input tensors have symbolic
+shapes. The compile engine will generate a dispatch function that invokes the
+specialized kernel first when the corresponding condition is met; otherwise,
+invoke the kernel that has no associated specialized condition (``dense_common``
+in this example). This part is still work in progress. More details will be
+provided after it is done.
+
+.. code:: python
+
+    def dense_strategy(attrs, inputs, out_type, target):
+        m = inputs[0].shape[0]
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_dense(dense_compute1),
+            wrap_topi_schedule(dense_schedule1),
+            name="dense_common")
+
+        with tvm.te.SpecializedCondition(m > 16):
+            strategy.add_implementation(
+                wrap_compute_dense(dense_compute2),
+                wrap_topi_schedule(dense_schedule2),
+                name="dense_for_large_m",
+                plevel=15)
+
+        return strategy
+
+
+Register strategy for a new target
+----------------------------------
+
+There are two ways to register strategies for a new target. The more
+straightforward one is adding a new target file in the directory
+``python/tvm/relay/op/strategy``. You only need to customize the strategy for
+ops that have been implemented for this new target and reuse the generic
+strategies for the rest.
+
+Alternatively, you can also register the strategy for the new target outside the
+TVM python library. The following code snippet shows an example how to do
+so. You can find more examples in ``vta/python/vta/top/op.py``.
+
+.. code:: python
+
+    @relay.op.strategy.someop_strategy.register("mytarget")
+    def someop_strategy_mytarget(attrs, inputs, out_type, target):
+        ...
+
+
+Select implementation from op strategy
 
 Review comment:
   ditto

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393454193
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393454735
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
+Similarly, we need to first define the ``FTVMStrategy`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @override_native_generic_func("my_new_op_strategy")
+    def my_new_op_strategy(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.my_new_op),
+            wrap_topi_schedule(topi.generic.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @my_new_op_strategy.register("cpu")
+    def my_new_op_strategy_cpu(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.x86.my_new_op),
+            wrap_topi_schedule(topi.x86.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
 
 Review comment:
   moved the introduction of API before this block.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393252292

Review comment:
Also for the priority level, I would add a pointer to the section in question, "Select implementation from op strategy"

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [incubator-tvm] tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393245105

Review comment:
Provide and example of what specialized conditions are applied here, perhaps datatype matching, or data layout etc.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [incubator-tvm] comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393848565
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,270 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to the implementations defined in TOPI
+library, a compute and schedule function need to be registered to each Relay
+operator.  However, compute and schedule functions are usually specialized for
+each target, and further, even for the same target, we may have multiple
+algorithms and implementations available. To deal with the complexity, we
+introduce operator strategy to allow developers to define a flexible lowering
+strategy for each operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the use of priority level is explained in
+`Select Implementation from Op Strategy`_).
+
+The ``OpStrategy`` includes a list of ``OpSpecialization``. Each ``OpSpecialization``
+contains a list of ``OpImplementation`` associated with a ``SpecializedCondition``
+(see definition in ``include/tvm/te/schedule.h``).  The ``SpecializedCondition``
+can be null, indicating the implementations are generally applicable;
+otherwise, the implementations are only considered when the specialized
+condition is satisfied. ``SpecializedCondition`` consists of a list
+of clauses defined in Tensor Expression in conjunctive normal form (CNF) and
+only supports conditions on tensor shapes.
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to.
+
+
+
+Register Strategy for An Operator
+---------------------------------
+
+There are three methods to register a strategy function for an operator,
 
 Review comment:
   Some thoughts to this section:
   - It's better to define "strategy function" first. Something like "a strategy function is used to determine which pair of compute and schedule function should be used by given a workload".
   
   - The third method described at the end of this section (`add_implement`) is not a method of registering a strategy function to me. Instead, it is more like how to implement a strategy function. It might be better to have a separate section called "Implement a Strategy for An Operator", or just change this section title to "Implement and Register a Strategy for An Operator".
   
   - My understanding is that the first approach is a simple one because we can reuse the compute function for those patterns. In this case, it might be better to switch the order of first and second methods. That is, we first introduce a method of implementing and registering a strategy function for dense. After that, we mention if your operator fits certain patterns you can simplify your work.
   
   - In the description of strategy function implementation, I think it might be better to mention `topk` only. As a result, in the beginning of the next section "Advanced Strategy Function", you can say some operators like dense require a more complicate strategy so it cannot be implemented like `topk`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393465730
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
+Similarly, we need to first define the ``FTVMStrategy`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @override_native_generic_func("my_new_op_strategy")
+    def my_new_op_strategy(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.my_new_op),
+            wrap_topi_schedule(topi.generic.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @my_new_op_strategy.register("cpu")
+    def my_new_op_strategy_cpu(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.x86.my_new_op),
+            wrap_topi_schedule(topi.x86.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+In this example, we use two wrapper function that wrap the topi compute and
+schedule function to conform with the required function signature. Usually we
+need to write a customized compute wrap function to retrieve different fields
+from op attributes. After that, we can then register this strategy to the new
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] tmoreau89 commented on issue #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

tmoreau89 commented on issue #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#issuecomment-601372080
 
 
   Nice this tutorial has better clarity now. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393248995
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
+Similarly, we need to first define the ``FTVMStrategy`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @override_native_generic_func("my_new_op_strategy")
+    def my_new_op_strategy(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.my_new_op),
+            wrap_topi_schedule(topi.generic.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @my_new_op_strategy.register("cpu")
+    def my_new_op_strategy_cpu(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.x86.my_new_op),
+            wrap_topi_schedule(topi.x86.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+In this example, we use two wrapper function that wrap the topi compute and
+schedule function to conform with the required function signature. Usually we
+need to write a customized compute wrap function to retrieve different fields
+from op attributes. After that, we can then register this strategy to the new
 
 Review comment:
   remove "then"

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393247092

Review comment:
Do specializations have a specific name?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [incubator-tvm] comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393397196
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
+Similarly, we need to first define the ``FTVMStrategy`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @override_native_generic_func("my_new_op_strategy")
+    def my_new_op_strategy(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.my_new_op),
+            wrap_topi_schedule(topi.generic.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @my_new_op_strategy.register("cpu")
+    def my_new_op_strategy_cpu(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.x86.my_new_op),
+            wrap_topi_schedule(topi.x86.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+In this example, we use two wrapper function that wrap the topi compute and
+schedule function to conform with the required function signature. Usually we
+need to write a customized compute wrap function to retrieve different fields
+from op attributes. After that, we can then register this strategy to the new
+operator by
+
+.. code:: python
+
+    register_strategy("my_new_op", strategy.my_new_op_strategy)
+
+
+Advanced strategy function
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The example above only shows the very basic strategy function.
+In this part, we will show a few advanced ways to define op strategy.
+
+First, we can add multiple implementations that use different algorithms to the
+same operator:
+
+.. code:: python
+
+    strategy.add_implementation(
+        wrap_compute_my_op(my_op_compute1),
+        wrap_topi_schedule(my_op_schedule1),
+        name="my_implementation1",
+        plevel=10)
+
+    if some_condition:
+        strategy.add_implementation(
+            wrap_compute_my_op(my_op_compute2),
+            wrap_topi_schedule(my_op_schedule2),
+            name="my_implementation2",
+            plevel=15)
+
+In this example, we add two implementations to the op strategy where
+implementation 2 is added with certain condition. ``my_implementation2`` will be
+used to compile this operator when ``some_condition`` is true as it has higher
+priority level (this could be changed if certain implementation is an AutoTVM
+template. See `Select implementation from op strategy`_ for more
+details). Otherwise, ``my_implementation1`` is used.
+
+We can extend the example above to third party library implementation. For
+example, we can add the implementation that invokes kernel in the third party
+library when the library is included in the target.
+
+.. code:: python
+
+    if "some_lib" in target.libs:
+        strategy.add_implementation(
+            wrap_compute_my_op(my_op_compute_lib),
+            wrap_topi_schedule(my_op_schedule_lib),
+            name="my_implementation_lib",
+            plevel=20)
+
+
+Further, we can add implementation specialized for a certain range of shapes.
+The code below shows an example of dense strategy that adds an implementation
+that is specialized for ``m`` greater than 16. The main difference between
+hardcode python condition like examples above and specialized condition is that
+it allows TVM to generate multiple kernels when the input tensors have symbolic
+shapes. The compile engine will generate a dispatch function that invokes the
+specialized kernel first when the corresponding condition is met; otherwise,
+invoke the kernel that has no associated specialized condition (``dense_common``
+in this example). This part is still work in progress. More details will be
+provided after it is done.
+
+.. code:: python
+
+    def dense_strategy(attrs, inputs, out_type, target):
+        m = inputs[0].shape[0]
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_dense(dense_compute1),
+            wrap_topi_schedule(dense_schedule1),
+            name="dense_common")
+
+        with tvm.te.SpecializedCondition(m > 16):
+            strategy.add_implementation(
+                wrap_compute_dense(dense_compute2),
+                wrap_topi_schedule(dense_schedule2),
+                name="dense_for_large_m",
+                plevel=15)
+
+        return strategy
+
+
+Register strategy for a new target
+----------------------------------
+
+There are two ways to register strategies for a new target. The more
+straightforward one is adding a new target file in the directory
+``python/tvm/relay/op/strategy``. You only need to customize the strategy for
+ops that have been implemented for this new target and reuse the generic
+strategies for the rest.
+
+Alternatively, you can also register the strategy for the new target outside the
+TVM python library. The following code snippet shows an example how to do
+so. You can find more examples in ``vta/python/vta/top/op.py``.
+
+.. code:: python
+
+    @relay.op.strategy.someop_strategy.register("mytarget")
+    def someop_strategy_mytarget(attrs, inputs, out_type, target):
+        ...
+
+
+Select implementation from op strategy
+--------------------------------------
+
+During the compilation, Relay compile engine needs to determine which
+implementation to use for an operator when there are multiple. The selection
+policy works as follows.
+
+When the input tensors to an operator or a fused op all have constant shapes,
+the compile engine first finds the best implementation based on AutoTVM tuning
+logs. If there is no implementation that is an AutoTVM template or all AutoTVM
+templates have fallback configs, the implementation with highest priority level
+will then be chosen. Implementations with same priority level in this case leads
+to an undefined behavior, and any of them might be selected.
 
 Review comment:
   I think it's fine tho. Throwing an error seems too serious to me (and users won't know what does that mean, because the error is for developers). If a developer assigns the same priority level to two implementations, op strategy selects either the one with better tuning result, or the one shows up earlier. It might be fine if we directly say "the first added implementation will be selected for the same priority level", but I am also fine with the current description since it implies we do not recommend developers to use the same priority level at all.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393164002
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
+Similarly, we need to first define the ``FTVMStrategy`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @override_native_generic_func("my_new_op_strategy")
+    def my_new_op_strategy(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.my_new_op),
+            wrap_topi_schedule(topi.generic.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @my_new_op_strategy.register("cpu")
+    def my_new_op_strategy_cpu(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.x86.my_new_op),
+            wrap_topi_schedule(topi.x86.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+In this example, we use two wrapper function that wrap the topi compute and
+schedule function to conform with the required function signature. Usually we
+need to write a customized compute wrap function to retrieve different fields
+from op attributes. After that, we can then register this strategy to the new
+operator by
+
+.. code:: python
+
+    register_strategy("my_new_op", strategy.my_new_op_strategy)
+
+
+Advanced strategy function
 
 Review comment:
   Capitalize each word for consistency.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393244452
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
 
 Review comment:
   a compute and schedule function need to be registered to each Relay operator

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on issue #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on issue #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#issuecomment-600867796
 
 
   I have thought about @comaniac comments and they make a lot of sense. I plan to separate the "Register Strategy for An Operator" into two parts, first describe how to define a strategy function, and second how to do the registration. I'll work on the update today. So let's keep the PR open for now.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393154822
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
 
 Review comment:
   s/implementation/the implementations

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393848565
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,270 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to the implementations defined in TOPI
+library, a compute and schedule function need to be registered to each Relay
+operator.  However, compute and schedule functions are usually specialized for
+each target, and further, even for the same target, we may have multiple
+algorithms and implementations available. To deal with the complexity, we
+introduce operator strategy to allow developers to define a flexible lowering
+strategy for each operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the use of priority level is explained in
+`Select Implementation from Op Strategy`_).
+
+The ``OpStrategy`` includes a list of ``OpSpecialization``. Each ``OpSpecialization``
+contains a list of ``OpImplementation`` associated with a ``SpecializedCondition``
+(see definition in ``include/tvm/te/schedule.h``).  The ``SpecializedCondition``
+can be null, indicating the implementations are generally applicable;
+otherwise, the implementations are only considered when the specialized
+condition is satisfied. ``SpecializedCondition`` consists of a list
+of clauses defined in Tensor Expression in conjunctive normal form (CNF) and
+only supports conditions on tensor shapes.
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to.
+
+
+
+Register Strategy for An Operator
+---------------------------------
+
+There are three methods to register a strategy function for an operator,
 
 Review comment:
   Some thoughts to this section:
   - It's better to define "strategy function" first. Something like "a strategy function is used to determine which pair of compute and schedule function should be used by given a workload".
   
   - The third method described at the end of this section (`add_implement`) is not a method of registering a strategy function to me. Instead, it is more like how to implement a strategy function. It might be better to have a separate section called "Implement a Strategy for An Operator", or just change this section title to "Implement and Register a Strategy for An Operator".
   
   - My understanding is that the first approach is a simple one because we can reuse the compute function for those patterns. In this case, it might be better to switch the order of first and second methods. That is, we first introduce a method of implementing and registering a strategy function for dense. After that, we mention if your operator fits certain patterns you can simply your work.
   
   - In the description of strategy function implementation, I think it might be better to mention `topk` only. As a result, in the beginning of the next section "Advanced Strategy Function", you can say some operators like dense require a more complicate strategy so it cannot be implemented like `topk`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393465376

Review comment:
fixed

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393454158
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
 
 Review comment:
   fixed
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393249109
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
+Similarly, we need to first define the ``FTVMStrategy`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @override_native_generic_func("my_new_op_strategy")
+    def my_new_op_strategy(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.my_new_op),
+            wrap_topi_schedule(topi.generic.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @my_new_op_strategy.register("cpu")
+    def my_new_op_strategy_cpu(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.x86.my_new_op),
+            wrap_topi_schedule(topi.x86.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+In this example, we use two wrapper function that wrap the topi compute and
+schedule function to conform with the required function signature. Usually we
+need to write a customized compute wrap function to retrieve different fields
+from op attributes. After that, we can then register this strategy to the new
+operator by
 
 Review comment:
   s/by/with

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393162407
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
+Similarly, we need to first define the ``FTVMStrategy`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @override_native_generic_func("my_new_op_strategy")
+    def my_new_op_strategy(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.my_new_op),
+            wrap_topi_schedule(topi.generic.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @my_new_op_strategy.register("cpu")
+    def my_new_op_strategy_cpu(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.x86.my_new_op),
+            wrap_topi_schedule(topi.x86.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+In this example, we use two wrapper function that wrap the topi compute and
 
 Review comment:
   s/two wrapper function/two wrapper functions

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393454046

Review comment:
Moved to the later part.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393465198
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
 
 Review comment:
   fixed

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393166193
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
+Similarly, we need to first define the ``FTVMStrategy`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @override_native_generic_func("my_new_op_strategy")
+    def my_new_op_strategy(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.my_new_op),
+            wrap_topi_schedule(topi.generic.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @my_new_op_strategy.register("cpu")
+    def my_new_op_strategy_cpu(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.x86.my_new_op),
+            wrap_topi_schedule(topi.x86.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+In this example, we use two wrapper function that wrap the topi compute and
+schedule function to conform with the required function signature. Usually we
+need to write a customized compute wrap function to retrieve different fields
+from op attributes. After that, we can then register this strategy to the new
+operator by
+
+.. code:: python
+
+    register_strategy("my_new_op", strategy.my_new_op_strategy)
+
+
+Advanced strategy function
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The example above only shows the very basic strategy function.
+In this part, we will show a few advanced ways to define op strategy.
+
+First, we can add multiple implementations that use different algorithms to the
+same operator:
+
+.. code:: python
+
+    strategy.add_implementation(
+        wrap_compute_my_op(my_op_compute1),
+        wrap_topi_schedule(my_op_schedule1),
+        name="my_implementation1",
+        plevel=10)
+
+    if some_condition:
+        strategy.add_implementation(
+            wrap_compute_my_op(my_op_compute2),
+            wrap_topi_schedule(my_op_schedule2),
+            name="my_implementation2",
+            plevel=15)
+
+In this example, we add two implementations to the op strategy where
+implementation 2 is added with certain condition. ``my_implementation2`` will be
 
 Review comment:
   s/certain condition/a certain condition

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393454298
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
+Similarly, we need to first define the ``FTVMStrategy`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @override_native_generic_func("my_new_op_strategy")
+    def my_new_op_strategy(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.my_new_op),
+            wrap_topi_schedule(topi.generic.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @my_new_op_strategy.register("cpu")
+    def my_new_op_strategy_cpu(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.x86.my_new_op),
+            wrap_topi_schedule(topi.x86.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+In this example, we use two wrapper function that wrap the topi compute and
+schedule function to conform with the required function signature. Usually we
+need to write a customized compute wrap function to retrieve different fields
+from op attributes. After that, we can then register this strategy to the new
+operator by
+
+.. code:: python
+
+    register_strategy("my_new_op", strategy.my_new_op_strategy)
+
+
+Advanced strategy function
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The example above only shows the very basic strategy function.
+In this part, we will show a few advanced ways to define op strategy.
+
+First, we can add multiple implementations that use different algorithms to the
+same operator:
+
+.. code:: python
+
+    strategy.add_implementation(
+        wrap_compute_my_op(my_op_compute1),
+        wrap_topi_schedule(my_op_schedule1),
+        name="my_implementation1",
+        plevel=10)
+
+    if some_condition:
+        strategy.add_implementation(
+            wrap_compute_my_op(my_op_compute2),
+            wrap_topi_schedule(my_op_schedule2),
+            name="my_implementation2",
+            plevel=15)
+
+In this example, we add two implementations to the op strategy where
+implementation 2 is added with certain condition. ``my_implementation2`` will be
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393251787
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
+Similarly, we need to first define the ``FTVMStrategy`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @override_native_generic_func("my_new_op_strategy")
+    def my_new_op_strategy(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.my_new_op),
+            wrap_topi_schedule(topi.generic.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @my_new_op_strategy.register("cpu")
+    def my_new_op_strategy_cpu(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.x86.my_new_op),
+            wrap_topi_schedule(topi.x86.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+In this example, we use two wrapper function that wrap the topi compute and
+schedule function to conform with the required function signature. Usually we
+need to write a customized compute wrap function to retrieve different fields
+from op attributes. After that, we can then register this strategy to the new
+operator by
+
+.. code:: python
+
+    register_strategy("my_new_op", strategy.my_new_op_strategy)
+
+
+Advanced strategy function
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The example above only shows the very basic strategy function.
+In this part, we will show a few advanced ways to define op strategy.
+
+First, we can add multiple implementations that use different algorithms to the
+same operator:
+
+.. code:: python
+
+    strategy.add_implementation(
+        wrap_compute_my_op(my_op_compute1),
+        wrap_topi_schedule(my_op_schedule1),
+        name="my_implementation1",
+        plevel=10)
+
+    if some_condition:
+        strategy.add_implementation(
+            wrap_compute_my_op(my_op_compute2),
+            wrap_topi_schedule(my_op_schedule2),
+            name="my_implementation2",
+            plevel=15)
+
+In this example, we add two implementations to the op strategy where
+implementation 2 is added with certain condition. ``my_implementation2`` will be
+used to compile this operator when ``some_condition`` is true as it has higher
+priority level (this could be changed if certain implementation is an AutoTVM
+template. See `Select implementation from op strategy`_ for more
+details). Otherwise, ``my_implementation1`` is used.
+
+We can extend the example above to third party library implementation. For
+example, we can add the implementation that invokes kernel in the third party
+library when the library is included in the target.
+
+.. code:: python
+
+    if "some_lib" in target.libs:
+        strategy.add_implementation(
+            wrap_compute_my_op(my_op_compute_lib),
+            wrap_topi_schedule(my_op_schedule_lib),
+            name="my_implementation_lib",
+            plevel=20)
+
+
+Further, we can add implementation specialized for a certain range of shapes.
+The code below shows an example of dense strategy that adds an implementation
+that is specialized for ``m`` greater than 16. The main difference between
+hardcode python condition like examples above and specialized condition is that
+it allows TVM to generate multiple kernels when the input tensors have symbolic
+shapes. The compile engine will generate a dispatch function that invokes the
+specialized kernel first when the corresponding condition is met; otherwise,
+invoke the kernel that has no associated specialized condition (``dense_common``
+in this example). This part is still work in progress. More details will be
+provided after it is done.
+
+.. code:: python
+
+    def dense_strategy(attrs, inputs, out_type, target):
+        m = inputs[0].shape[0]
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_dense(dense_compute1),
+            wrap_topi_schedule(dense_schedule1),
+            name="dense_common")
+
+        with tvm.te.SpecializedCondition(m > 16):
+            strategy.add_implementation(
+                wrap_compute_dense(dense_compute2),
+                wrap_topi_schedule(dense_schedule2),
+                name="dense_for_large_m",
+                plevel=15)
+
+        return strategy
+
+
+Register strategy for a new target
+----------------------------------
+
+There are two ways to register strategies for a new target. The more
+straightforward one is adding a new target file in the directory
+``python/tvm/relay/op/strategy``. You only need to customize the strategy for
+ops that have been implemented for this new target and reuse the generic
+strategies for the rest.
+
+Alternatively, you can also register the strategy for the new target outside the
+TVM python library. The following code snippet shows an example how to do
+so. You can find more examples in ``vta/python/vta/top/op.py``.
+
+.. code:: python
+
+    @relay.op.strategy.someop_strategy.register("mytarget")
+    def someop_strategy_mytarget(attrs, inputs, out_type, target):
+        ...
+
+
+Select implementation from op strategy
+--------------------------------------
+
+During the compilation, Relay compile engine needs to determine which
+implementation to use for an operator when there are multiple. The selection
+policy works as follows.
+
+When the input tensors to an operator or a fused op all have constant shapes,
+the compile engine first finds the best implementation based on AutoTVM tuning
+logs. If there is no implementation that is an AutoTVM template or all AutoTVM
+templates have fallback configs, the implementation with highest priority level
+will then be chosen. Implementations with same priority level in this case leads
+to an undefined behavior, and any of them might be selected.
 
 Review comment:
   this is not related to this PR, but shouldn't this undefined behavior trigger an error?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393454242
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
+Similarly, we need to first define the ``FTVMStrategy`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @override_native_generic_func("my_new_op_strategy")
+    def my_new_op_strategy(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.my_new_op),
+            wrap_topi_schedule(topi.generic.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @my_new_op_strategy.register("cpu")
+    def my_new_op_strategy_cpu(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.x86.my_new_op),
+            wrap_topi_schedule(topi.x86.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+In this example, we use two wrapper function that wrap the topi compute and
+schedule function to conform with the required function signature. Usually we
+need to write a customized compute wrap function to retrieve different fields
+from op attributes. After that, we can then register this strategy to the new
+operator by
+
+.. code:: python
+
+    register_strategy("my_new_op", strategy.my_new_op_strategy)
+
+
+Advanced strategy function
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393454109
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
 
 Review comment:
   fixed

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on issue #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on issue #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#issuecomment-601002936
 
 
   made an update. @comaniac @tmoreau89 could you take a look again?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393465828
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
+Similarly, we need to first define the ``FTVMStrategy`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @override_native_generic_func("my_new_op_strategy")
+    def my_new_op_strategy(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.my_new_op),
+            wrap_topi_schedule(topi.generic.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @my_new_op_strategy.register("cpu")
+    def my_new_op_strategy_cpu(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.x86.my_new_op),
+            wrap_topi_schedule(topi.x86.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+In this example, we use two wrapper function that wrap the topi compute and
+schedule function to conform with the required function signature. Usually we
+need to write a customized compute wrap function to retrieve different fields
+from op attributes. After that, we can then register this strategy to the new
+operator by
+
+.. code:: python
+
+    register_strategy("my_new_op", strategy.my_new_op_strategy)
+
+
+Advanced strategy function
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The example above only shows the very basic strategy function.
+In this part, we will show a few advanced ways to define op strategy.
+
+First, we can add multiple implementations that use different algorithms to the
+same operator:
+
+.. code:: python
+
+    strategy.add_implementation(
+        wrap_compute_my_op(my_op_compute1),
+        wrap_topi_schedule(my_op_schedule1),
+        name="my_implementation1",
+        plevel=10)
+
+    if some_condition:
+        strategy.add_implementation(
+            wrap_compute_my_op(my_op_compute2),
+            wrap_topi_schedule(my_op_schedule2),
+            name="my_implementation2",
+            plevel=15)
+
+In this example, we add two implementations to the op strategy where
+implementation 2 is added with certain condition. ``my_implementation2`` will be
+used to compile this operator when ``some_condition`` is true as it has higher
+priority level (this could be changed if certain implementation is an AutoTVM
+template. See `Select implementation from op strategy`_ for more
+details). Otherwise, ``my_implementation1`` is used.
+
+We can extend the example above to third party library implementation. For
+example, we can add the implementation that invokes kernel in the third party
+library when the library is included in the target.
+
+.. code:: python
+
+    if "some_lib" in target.libs:
+        strategy.add_implementation(
+            wrap_compute_my_op(my_op_compute_lib),
+            wrap_topi_schedule(my_op_schedule_lib),
+            name="my_implementation_lib",
+            plevel=20)
+
+
+Further, we can add implementation specialized for a certain range of shapes.
+The code below shows an example of dense strategy that adds an implementation
+that is specialized for ``m`` greater than 16. The main difference between
+hardcode python condition like examples above and specialized condition is that
+it allows TVM to generate multiple kernels when the input tensors have symbolic
+shapes. The compile engine will generate a dispatch function that invokes the
+specialized kernel first when the corresponding condition is met; otherwise,
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393454512
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
+Similarly, we need to first define the ``FTVMStrategy`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @override_native_generic_func("my_new_op_strategy")
+    def my_new_op_strategy(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.my_new_op),
+            wrap_topi_schedule(topi.generic.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @my_new_op_strategy.register("cpu")
+    def my_new_op_strategy_cpu(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.x86.my_new_op),
+            wrap_topi_schedule(topi.x86.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+In this example, we use two wrapper function that wrap the topi compute and
+schedule function to conform with the required function signature. Usually we
+need to write a customized compute wrap function to retrieve different fields
+from op attributes. After that, we can then register this strategy to the new
+operator by
+
+.. code:: python
+
+    register_strategy("my_new_op", strategy.my_new_op_strategy)
+
+
+Advanced strategy function
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The example above only shows the very basic strategy function.
+In this part, we will show a few advanced ways to define op strategy.
+
+First, we can add multiple implementations that use different algorithms to the
+same operator:
+
+.. code:: python
+
+    strategy.add_implementation(
+        wrap_compute_my_op(my_op_compute1),
+        wrap_topi_schedule(my_op_schedule1),
+        name="my_implementation1",
+        plevel=10)
+
+    if some_condition:
+        strategy.add_implementation(
+            wrap_compute_my_op(my_op_compute2),
+            wrap_topi_schedule(my_op_schedule2),
+            name="my_implementation2",
+            plevel=15)
+
+In this example, we add two implementations to the op strategy where
+implementation 2 is added with certain condition. ``my_implementation2`` will be
+used to compile this operator when ``some_condition`` is true as it has higher
+priority level (this could be changed if certain implementation is an AutoTVM
+template. See `Select implementation from op strategy`_ for more
+details). Otherwise, ``my_implementation1`` is used.
+
+We can extend the example above to third party library implementation. For
+example, we can add the implementation that invokes kernel in the third party
+library when the library is included in the target.
+
+.. code:: python
+
+    if "some_lib" in target.libs:
+        strategy.add_implementation(
+            wrap_compute_my_op(my_op_compute_lib),
+            wrap_topi_schedule(my_op_schedule_lib),
+            name="my_implementation_lib",
+            plevel=20)
+
+
+Further, we can add implementation specialized for a certain range of shapes.
+The code below shows an example of dense strategy that adds an implementation
+that is specialized for ``m`` greater than 16. The main difference between
+hardcode python condition like examples above and specialized condition is that
+it allows TVM to generate multiple kernels when the input tensors have symbolic
+shapes. The compile engine will generate a dispatch function that invokes the
+specialized kernel first when the corresponding condition is met; otherwise,
+invoke the kernel that has no associated specialized condition (``dense_common``
+in this example). This part is still work in progress. More details will be
+provided after it is done.
+
+.. code:: python
+
+    def dense_strategy(attrs, inputs, out_type, target):
+        m = inputs[0].shape[0]
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_dense(dense_compute1),
+            wrap_topi_schedule(dense_schedule1),
+            name="dense_common")
+
+        with tvm.te.SpecializedCondition(m > 16):
+            strategy.add_implementation(
+                wrap_compute_dense(dense_compute2),
+                wrap_topi_schedule(dense_schedule2),
+                name="dense_for_large_m",
+                plevel=15)
+
+        return strategy
+
+
+Register strategy for a new target
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

comaniac commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393160512
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
 
 Review comment:
   s/same/the same

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393465667

Review comment:
Updated

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393453643
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
 
 Review comment:
   fixed

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r394635152
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
+Similarly, we need to first define the ``FTVMStrategy`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @override_native_generic_func("my_new_op_strategy")
+    def my_new_op_strategy(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.my_new_op),
+            wrap_topi_schedule(topi.generic.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @my_new_op_strategy.register("cpu")
+    def my_new_op_strategy_cpu(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.x86.my_new_op),
+            wrap_topi_schedule(topi.x86.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+In this example, we use two wrapper function that wrap the topi compute and
+schedule function to conform with the required function signature. Usually we
+need to write a customized compute wrap function to retrieve different fields
+from op attributes. After that, we can then register this strategy to the new
+operator by
+
+.. code:: python
+
+    register_strategy("my_new_op", strategy.my_new_op_strategy)
+
+
+Advanced strategy function
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The example above only shows the very basic strategy function.
+In this part, we will show a few advanced ways to define op strategy.
+
+First, we can add multiple implementations that use different algorithms to the
+same operator:
+
+.. code:: python
+
+    strategy.add_implementation(
+        wrap_compute_my_op(my_op_compute1),
+        wrap_topi_schedule(my_op_schedule1),
+        name="my_implementation1",
+        plevel=10)
+
+    if some_condition:
+        strategy.add_implementation(
+            wrap_compute_my_op(my_op_compute2),
+            wrap_topi_schedule(my_op_schedule2),
+            name="my_implementation2",
+            plevel=15)
+
+In this example, we add two implementations to the op strategy where
+implementation 2 is added with certain condition. ``my_implementation2`` will be
+used to compile this operator when ``some_condition`` is true as it has higher
+priority level (this could be changed if certain implementation is an AutoTVM
+template. See `Select implementation from op strategy`_ for more
+details). Otherwise, ``my_implementation1`` is used.
+
+We can extend the example above to third party library implementation. For
+example, we can add the implementation that invokes kernel in the third party
+library when the library is included in the target.
+
+.. code:: python
+
+    if "some_lib" in target.libs:
+        strategy.add_implementation(
+            wrap_compute_my_op(my_op_compute_lib),
+            wrap_topi_schedule(my_op_schedule_lib),
+            name="my_implementation_lib",
+            plevel=20)
+
+
+Further, we can add implementation specialized for a certain range of shapes.
+The code below shows an example of dense strategy that adds an implementation
+that is specialized for ``m`` greater than 16. The main difference between
+hardcode python condition like examples above and specialized condition is that
+it allows TVM to generate multiple kernels when the input tensors have symbolic
+shapes. The compile engine will generate a dispatch function that invokes the
+specialized kernel first when the corresponding condition is met; otherwise,
+invoke the kernel that has no associated specialized condition (``dense_common``
+in this example). This part is still work in progress. More details will be
+provided after it is done.
+
+.. code:: python
+
+    def dense_strategy(attrs, inputs, out_type, target):
+        m = inputs[0].shape[0]
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_dense(dense_compute1),
+            wrap_topi_schedule(dense_schedule1),
+            name="dense_common")
+
+        with tvm.te.SpecializedCondition(m > 16):
+            strategy.add_implementation(
+                wrap_compute_dense(dense_compute2),
+                wrap_topi_schedule(dense_schedule2),
+                name="dense_for_large_m",
+                plevel=15)
+
+        return strategy
+
+
+Register strategy for a new target
+----------------------------------
+
+There are two ways to register strategies for a new target. The more
+straightforward one is adding a new target file in the directory
+``python/tvm/relay/op/strategy``. You only need to customize the strategy for
+ops that have been implemented for this new target and reuse the generic
+strategies for the rest.
+
+Alternatively, you can also register the strategy for the new target outside the
+TVM python library. The following code snippet shows an example how to do
+so. You can find more examples in ``vta/python/vta/top/op.py``.
+
+.. code:: python
+
+    @relay.op.strategy.someop_strategy.register("mytarget")
+    def someop_strategy_mytarget(attrs, inputs, out_type, target):
+        ...
+
+
+Select implementation from op strategy
+--------------------------------------
+
+During the compilation, Relay compile engine needs to determine which
+implementation to use for an operator when there are multiple. The selection
+policy works as follows.
+
+When the input tensors to an operator or a fused op all have constant shapes,
+the compile engine first finds the best implementation based on AutoTVM tuning
+logs. If there is no implementation that is an AutoTVM template or all AutoTVM
+templates have fallback configs, the implementation with highest priority level
+will then be chosen. Implementations with same priority level in this case leads
+to an undefined behavior, and any of them might be selected.
 
 Review comment:
   Thanks, that makes sense

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] tmoreau89 commented on issue #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

tmoreau89 commented on issue #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#issuecomment-601372494
 
 
   Thanks @icemelon9 @comaniac the PR has been merged.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393465112
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
+Similarly, we need to first define the ``FTVMStrategy`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @override_native_generic_func("my_new_op_strategy")
+    def my_new_op_strategy(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.my_new_op),
+            wrap_topi_schedule(topi.generic.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @my_new_op_strategy.register("cpu")
+    def my_new_op_strategy_cpu(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.x86.my_new_op),
+            wrap_topi_schedule(topi.x86.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+In this example, we use two wrapper function that wrap the topi compute and
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

tmoreau89 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393244037

Review comment:
s/usability/use

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [incubator-tvm] icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy

Posted by GitBox <gi...@apache.org>.

icemelon9 commented on a change in pull request #5078: [DOC] Add doc for Relay op strategy
URL: https://github.com/apache/incubator-tvm/pull/5078#discussion_r393465800
 
 

 ##########
 File path: docs/dev/relay_op_strategy.rst
 ##########
 @@ -0,0 +1,256 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _relay-op-strategy:
+
+Relay Operator Strategy
+=======================
+
+In order to lower Relay operators to implementation defined in TOPI library, the
+compute and schedule functions need to be registered to Relay operators.
+However, compute and schedule functions are usually specialized for each target,
+and further, even for the same target, we may have multiple algorithms and
+implementations available. To deal with the complexity, we introduce operator
+strategy to allow developers to define a flexible lowering strategy for each
+operator and target.
+
+
+Operator Strategy Design
+------------------------
+
+The basic element in operator strategy is an ``OpImplementation``. It includes
+the a pair of compute and schedule function, the name of the implementation,
+and a priority level (the usability of priority level will be explained below).
+
+The ``OpStrategy`` includes a list of specializations. Each specialization
+contains a list of ``OpImplementation`` associated with a specialized condition
+(see ``SpecializedCondition`` definition in ``include/tvm/te/schedule.h``).  The
+specialized condition can be null, indicating the implementations are generally
+applicable; otherwise, the implementations should only be used when the
+specialized condition is satisfied. ``OpStrategy`` provides only one API,
+adding an implementation to the strategy:
+
+.. code:: python
+
+    def add_implementation(self, compute, schedule, name="default", plevel=10)
+
+Last, a ``FTVMStrategy`` function is registered to each Relay operator.
+``FTVMStrategy`` is a generic function (see ``include/tvm/target/generic_func.h``),
+that can be overwritten for each target. The function signature is
+
+.. code:: c
+
+    OpStrategy(const Attrs& attrs, const Array<Tensor>& inputs, const Type& out_type, const Target& target)
+
+, that the function returns an ``OpStrategy`` given the op attributes, input
+tensors, output types, and target to compile to,
+
+
+
+Register strategy for a new operator
+------------------------------------
+
+There are three methods to register a strategy function for an operator,
+defined in ``python/tvm/relay/op/op.py``.
+
+First, for operators that have injective, broadcast, or reduction pattern, we
+can call ``register_injective_schedule``, ``register_broadcast_schedule``, and
+``register_reduce_schedule`` repsectively. The schedule function for these
+patterns are already registered by each target and can be applied to these
+operators. We assume the compute function should be same across all targets, and
+``FTVMCompute`` needs to be registered to the op before invoking register
+schedule.
+
+.. code:: python
+
+    register_injective_schedule("my_new_op")
+
+Second, for operators that doesn't have these common patterns mentioned before,
+but also have the same compute function for all targets, we can use
+``register_schedule`` API. But before that, we need to first define the
+``FTVMSchedule`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @generic_func
+    def schedule_my_new_op(attrs, outs, target):
+        ...
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @schedule_my_new_op.register("cpu")
+    def schedule_my_new_op_cpu(attrs, outs, target):
+        ...
+
+Now that we've created the ``FTVMSchedule`` for this new operator, we can
+register the strategy using ``register_schedule``:
+
+.. code:: python
+
+    register_schedule("my_new_op", strategy.schedule_my_new_op)
+
+Third, for most comprehensive usage of op strategy, we can allow operator to use
+different implementation for both compute and schedule for different targets.
+Similarly, we need to first define the ``FTVMStrategy`` function as follows:
+
+.. code:: python
+
+    # add to python/tvm/relay/op/strategy/generic.py
+    @override_native_generic_func("my_new_op_strategy")
+    def my_new_op_strategy(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.my_new_op),
+            wrap_topi_schedule(topi.generic.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+    # add to each target file in python/tvm/relay/op/strategy, e.g., x86.py, cuda.py, etc.
+    @my_new_op_strategy.register("cpu")
+    def my_new_op_strategy_cpu(attrs, inputs, out_type, target):
+        strategy = _op.OpStrategy()
+        strategy.add_implementation(
+            wrap_compute_my_new_op(topi.x86.my_new_op),
+            wrap_topi_schedule(topi.x86.schedule_my_new_op),
+            name="my_new_op.generic")
+        return strategy
+
+In this example, we use two wrapper function that wrap the topi compute and
+schedule function to conform with the required function signature. Usually we
+need to write a customized compute wrap function to retrieve different fields
+from op attributes. After that, we can then register this strategy to the new
+operator by
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services