You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2020/03/11 04:58:27 UTC
[GitHub] [incubator-tvm] hypercubestart opened a new pull request #5039:
GradientCell Relay Pass
hypercubestart opened a new pull request #5039: GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039
Add GradientCell relay pass which introduces the GradCell datatype. This pass can delay memory allocation and potentially improve memory usage and performance by delaying instantiations of zero-filled/one-filled tensors until necessary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] hypercubestart commented on a change in pull
request #5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
hypercubestart commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r396838303
##########
File path: python/tvm/relay/transform/transform.py
##########
@@ -219,6 +219,18 @@ def DeadCodeElimination(inline_once=False):
"""
return _ffi_api.DeadCodeElimination(inline_once)
+def GradientCell():
Review comment:
renamed to `LazyGradientInit`, highlight that this pass should be used after reverse mode ad to lazily initiate tensor gradients
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] MarisaKirisame commented on a change in pull
request #5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
MarisaKirisame commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r391388236
##########
File path: tests/python/relay/test_pass_gradient_cell.py
##########
@@ -0,0 +1,384 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay import create_executor, transform
+from tvm.relay.testing import rand, run_infer_type
+from tvm.testing import assert_allclose
+import pytest
+
+def grad_cell_type(mod, shape, dtype):
+ grad_type = mod.get_global_type_var("GradCell")
+ type_arg = relay.TensorType(shape, dtype)
+ return grad_type(type_arg)
+
+def test_add():
+ mod = tvm.IRModule()
+ mod.import_from_std("gradient.rly")
Review comment:
a better design will be to import_from_std inside the transform.
If it was import before it will do nothing, so it is safe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] hypercubestart commented on a change in pull
request #5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
hypercubestart commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r396838408
##########
File path: src/relay/transforms/gradient_cell.cc
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ *
+ * \file gradient_cell.cc
+ *
+ * \brief Convert all tensors to a Gradient Cell
+ *
+ * This pass delays or removes memory allocation by converting tensors into
+ * GradCell, an algebraic data type defined in gradient.rly
+ *
+ * This will delay or decrease memory usage. All calls to
+ * ones, ones_like, zeros, zeros_like will call the One or Zero constructor
+ * of GradCell, which will not instantiate in memory until needed. All other cases result
+ * in using the Raw constructor which means the tensor is instantiated in memory.
+ *
+ * It also overloads + and * operation which can increase performance when doing
+ * operations involving tensors with values of only 0 or 1.
+ *
+ * Note: this pass can only be used with functions where the input/output types are
+ * a combination of TupleTypes and TensorTypes
+ *
+ * This pass optimizes 6 ops:
+ * - add
+ * - multiply
+ * - ones
+ * - ones_like
+ * - zeros
+ * - zeros_like
+ *
+ * This pass makes use of three visitor. The most important one visits the entire function,
+ * one is used for wrap inputs and one to unwrap outputs.
+ *
+ * For example:
+ * fn: TensorType[(10,10), float32] -> TensorType[(10,10), float32]
+ *
+ * After this pass
+ * fn: GradCell[TensorType[(10,10), float32]] -> GradCell[TensorType[(10,10), float32]]
+ *
+ * Thus, it is necessary to wrap this outer function so that the input/output types remain the same
+ */
+
+#include <tvm/relay/analysis.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/ir/type_functor.h>
+#include <tvm/relay/transform.h>
+#include "let_list.h"
+
+namespace tvm {
+namespace relay {
+
+/*!
+* \brief Visitor to wrap inputs
+*/
+class InputVisitor: public ExprFunctor<Expr(const Expr&, const Type&)> {
+ public:
+ explicit InputVisitor(IRModule module): module_(module) {}
+
+ Expr wrapExpr(const Expr expr, const Type& type) {
+ if (type.as<TensorTypeNode>()) {
+ return Call(module_->GetConstructor("GradCell", "Raw"),
+ {expr}, Attrs(), {type});
+ } else if (auto* type_anno = type.as<TupleTypeNode>()) {
+ tvm::Array<Expr> fields;
+ for (size_t i = 0; i < type_anno->fields.size(); i++) {
+ const Type& t = type_anno->fields[i];
+ fields.push_back(this->VisitExpr(TupleGetItem(expr, i), t));
+ }
+ Expr tuple = Tuple(fields);
+ return tuple;
+ }
+
+ return expr;
+ }
+
+ Expr VisitExpr_(const VarNode* op, const Type& t) final {
+ std::cout << op->type_annotation << std::endl;
+ return wrapExpr(GetRef<Var>(op), op->type_annotation);
+ }
+
+ Expr VisitExpr_(const TupleGetItemNode* op, const Type& t) final {
+ return wrapExpr(GetRef<TupleGetItem>(op), t);
+ }
+ private:
+ IRModule module_;
+};
+
+/*!
+* \brief Visitor to unwrap output
Review comment:
done thanks for review!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] MarisaKirisame commented on a change in pull
request #5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
MarisaKirisame commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r396597422
##########
File path: src/relay/transforms/gradient_cell.cc
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ *
+ * \file gradient_cell.cc
+ *
+ * \brief Convert all tensors to a Gradient Cell
+ *
+ * This pass delays or removes memory allocation by converting tensors into
+ * GradCell, an algebraic data type defined in gradient.rly
+ *
+ * This will delay or decrease memory usage. All calls to
+ * ones, ones_like, zeros, zeros_like will call the One or Zero constructor
+ * of GradCell, which will not instantiate in memory until needed. All other cases result
+ * in using the Raw constructor which means the tensor is instantiated in memory.
+ *
+ * It also overloads + and * operation which can increase performance when doing
+ * operations involving tensors with values of only 0 or 1.
+ *
+ * Note: this pass can only be used with functions where the input/output types are
+ * a combination of TupleTypes and TensorTypes
+ *
+ * This pass optimizes 6 ops:
+ * - add
+ * - multiply
+ * - ones
+ * - ones_like
+ * - zeros
+ * - zeros_like
+ *
+ * This pass makes use of three visitor. The most important one visits the entire function,
+ * one is used for wrap inputs and one to unwrap outputs.
+ *
+ * For example:
+ * fn: TensorType[(10,10), float32] -> TensorType[(10,10), float32]
+ *
+ * After this pass
+ * fn: GradCell[TensorType[(10,10), float32]] -> GradCell[TensorType[(10,10), float32]]
+ *
+ * Thus, it is necessary to wrap this outer function so that the input/output types remain the same
+ */
+
+#include <tvm/relay/analysis.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/ir/type_functor.h>
+#include <tvm/relay/transform.h>
+#include "let_list.h"
+
+namespace tvm {
+namespace relay {
+
+/*!
+* \brief Get constructor of GradCell TypeDef with name_hint
+*
+* module must have TypeDefinition of GradCell (defined in gradient.rly)
+*/
+Constructor getGradCellConstructor(IRModule module, std::string name_hint) {
+ TypeData gradCell = module->LookupTypeDef("GradCell");
Review comment:
can you refactor this function into module?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] hypercubestart commented on a change in pull
request #5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
hypercubestart commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r396838434
##########
File path: src/relay/transforms/gradient_cell.cc
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ *
+ * \file gradient_cell.cc
+ *
+ * \brief Convert all tensors to a Gradient Cell
+ *
+ * This pass delays or removes memory allocation by converting tensors into
+ * GradCell, an algebraic data type defined in gradient.rly
+ *
+ * This will delay or decrease memory usage. All calls to
+ * ones, ones_like, zeros, zeros_like will call the One or Zero constructor
+ * of GradCell, which will not instantiate in memory until needed. All other cases result
+ * in using the Raw constructor which means the tensor is instantiated in memory.
+ *
+ * It also overloads + and * operation which can increase performance when doing
+ * operations involving tensors with values of only 0 or 1.
+ *
+ * Note: this pass can only be used with functions where the input/output types are
+ * a combination of TupleTypes and TensorTypes
+ *
+ * This pass optimizes 6 ops:
+ * - add
+ * - multiply
+ * - ones
+ * - ones_like
+ * - zeros
+ * - zeros_like
+ *
+ * This pass makes use of three visitor. The most important one visits the entire function,
+ * one is used for wrap inputs and one to unwrap outputs.
+ *
+ * For example:
+ * fn: TensorType[(10,10), float32] -> TensorType[(10,10), float32]
+ *
+ * After this pass
+ * fn: GradCell[TensorType[(10,10), float32]] -> GradCell[TensorType[(10,10), float32]]
+ *
+ * Thus, it is necessary to wrap this outer function so that the input/output types remain the same
+ */
+
+#include <tvm/relay/analysis.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/ir/type_functor.h>
+#include <tvm/relay/transform.h>
+#include "let_list.h"
+
+namespace tvm {
+namespace relay {
+
+/*!
+* \brief Visitor to wrap inputs
+*/
+class InputVisitor: public ExprFunctor<Expr(const Expr&, const Type&)> {
+ public:
+ explicit InputVisitor(IRModule module): module_(module) {}
+
+ Expr wrapExpr(const Expr expr, const Type& type) {
+ if (type.as<TensorTypeNode>()) {
+ return Call(module_->GetConstructor("GradCell", "Raw"),
+ {expr}, Attrs(), {type});
+ } else if (auto* type_anno = type.as<TupleTypeNode>()) {
+ tvm::Array<Expr> fields;
+ for (size_t i = 0; i < type_anno->fields.size(); i++) {
+ const Type& t = type_anno->fields[i];
+ fields.push_back(this->VisitExpr(TupleGetItem(expr, i), t));
+ }
+ Expr tuple = Tuple(fields);
+ return tuple;
+ }
+
+ return expr;
+ }
+
+ Expr VisitExpr_(const VarNode* op, const Type& t) final {
+ std::cout << op->type_annotation << std::endl;
+ return wrapExpr(GetRef<Var>(op), op->type_annotation);
+ }
+
+ Expr VisitExpr_(const TupleGetItemNode* op, const Type& t) final {
+ return wrapExpr(GetRef<TupleGetItem>(op), t);
+ }
+ private:
+ IRModule module_;
+};
+
+/*!
+* \brief Visitor to unwrap output
+*/
+class OutputVisitor: public ExprFunctor<Expr(const Expr&, const Type&)> {
+ public:
+ explicit OutputVisitor(IRModule module): module_(module) {}
+
+ Expr unwrapExpr(const Expr expr, const Type& type) {
Review comment:
done thanks for review!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] jroesch merged pull request #5039: [Relay]
GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
jroesch merged pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] MarisaKirisame commented on a change in pull
request #5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
MarisaKirisame commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r396596815
##########
File path: src/relay/transforms/gradient_cell.cc
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ *
+ * \file gradient_cell.cc
+ *
+ * \brief Convert all tensors to a Gradient Cell
+ *
+ * This pass delays or removes memory allocation by converting tensors into
+ * GradCell, an algebraic data type defined in gradient.rly
+ *
+ * This will delay or decrease memory usage. All calls to
+ * ones, ones_like, zeros, zeros_like will call the One or Zero constructor
+ * of GradCell, which will not instantiate in memory until needed. All other cases result
+ * in using the Raw constructor which means the tensor is instantiated in memory.
+ *
+ * It also overloads + and * operation which can increase performance when doing
+ * operations involving tensors with values of only 0 or 1.
+ *
+ * Note: this pass can only be used with functions where the input/output types are
+ * a combination of TupleTypes and TensorTypes
+ *
+ * This pass optimizes 6 ops:
+ * - add
+ * - multiply
+ * - ones
+ * - ones_like
+ * - zeros
+ * - zeros_like
+ *
+ * This pass makes use of three visitor. The most important one visits the entire function,
+ * one is used for wrap inputs and one to unwrap outputs.
+ *
+ * For example:
+ * fn: TensorType[(10,10), float32] -> TensorType[(10,10), float32]
+ *
+ * After this pass
+ * fn: GradCell[TensorType[(10,10), float32]] -> GradCell[TensorType[(10,10), float32]]
+ *
+ * Thus, it is necessary to wrap this outer function so that the input/output types remain the same
+ */
+
+#include <tvm/relay/analysis.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/ir/type_functor.h>
+#include <tvm/relay/transform.h>
+#include "let_list.h"
+
+namespace tvm {
+namespace relay {
+
+/*!
+* \brief Get constructor of GradCell TypeDef with name_hint
+*
+* module must have TypeDefinition of GradCell (defined in gradient.rly)
+*/
+Constructor getGradCellConstructor(IRModule module, std::string name_hint) {
+ TypeData gradCell = module->LookupTypeDef("GradCell");
+ for (Constructor c : gradCell->constructors) {
+ if (name_hint.compare(c->name_hint) == 0) {
+ return c;
+ }
+ }
+
+ LOG(FATAL) << "Constructor " << name_hint << "not found in GradCell typedata.";
+ throw std::runtime_error("Constructor not found in GradCell typedata");
+}
+
+/*!
+* \brief Visitor to wrap inputs
+*/
+class InputVisitor: public ExprFunctor<Expr(const Expr&, const Type&)> {
+ public:
+ explicit InputVisitor(IRModule module): module_(module) {}
+
+ Expr wrapExpr(const Expr expr, const Type& type) {
+ if (type.as<TensorTypeNode>()) {
+ return CallNode::make(getGradCellConstructor(module_, "Raw"),
+ {expr}, Attrs(), {type});
+ } else if (auto* type_anno = type.as<TupleTypeNode>()) {
+ tvm::Array<Expr> fields;
+ for (size_t i = 0; i < type_anno->fields.size(); i++) {
+ const Type& t = type_anno->fields[i];
+ fields.push_back(this->VisitExpr(TupleGetItemNode::make(expr, i), t));
+ }
+ Expr tuple = TupleNode::make(fields);
+ return tuple;
+ }
+
+ return expr;
+ }
+
+ Expr VisitExpr_(const VarNode* op, const Type& t) final {
+ std::cout << op->type_annotation << std::endl;
+ return wrapExpr(GetRef<Var>(op), op->type_annotation);
+ }
+
+ Expr VisitExpr_(const TupleGetItemNode* op, const Type& t) final {
+ return wrapExpr(GetRef<TupleGetItem>(op), t);
+ }
+ private:
+ IRModule module_;
+};
+
+/*!
+* \brief Visitor to unwrap output
+*/
+class OutputVisitor: public ExprFunctor<Expr(const Expr&, const Type&)> {
+ public:
+ explicit OutputVisitor(IRModule module): module_(module) {}
+
+ Expr unwrapExpr(const Expr expr, const Type& type) {
+ if (auto* type_call = type.as<TypeCallNode>()) {
+ if (type_call->func.same_as(module_->GetGlobalTypeVar("GradCell"))) {
+ return CallNode::make(module_->GetGlobalVar("FromGradCell"), {expr});
+ }
+ return expr;
+ } else if (auto* type_anno = type.as<TupleTypeNode>()) {
+ tvm::Array<Expr> fields;
+ for (size_t i = 0; i < type_anno->fields.size(); i++) {
+ const Type& t = type_anno->fields[i];
+ fields.push_back(this->VisitExpr(TupleGetItemNode::make(expr, i), t));
+ }
+ Expr tuple = TupleNode::make(fields);
+ return tuple;
+ }
+
+ return expr;
+ }
+
+ Expr VisitExpr_(const CallNode* op, const Type& t) final {
+ return unwrapExpr(GetRef<Call>(op), t);
+ }
+
+ Expr VisitExpr_(const TupleGetItemNode* op, const Type& t) final {
+ return unwrapExpr(GetRef<TupleGetItem>(op), t);
+ }
+ private:
+ IRModule module_;
+};
+
+class GradientCellTransform: public ExprMutator, public TypeMutator {
+ public:
+ explicit GradientCellTransform(IRModule module):
+ module_(module) {
+ module_->ImportFromStd("gradient.rly");
+ }
+
+ /*!
+ * \brief apply GradientCell transformation and wrap function
+ * so that function type stays the same
+ *
+ * input/output types should only be a combination of TupleTypes and TensorTypes
+ */
+ Expr transform(const Expr& e) {
+ auto* f = (e).as<FunctionNode>();
+ auto* transformed = this->Mutate(e).as<FunctionNode>();
+
+ if (e.same_as(GetRef<Function>(transformed))) {
+ return GetRef<Function>(transformed);
+ }
+
+ // wrap inputs of Tensor type using InputVisitor class
+ tvm::Array<Expr> args;
+ for (Var var : f->params) {
+ Expr wrappedInput = InputVisitor(module_).VisitExpr(var, var->checked_type());
+ args.push_back(wrappedInput);
+ }
+ Expr transformedExpr = CallNode::make(GetRef<Function>(transformed), args);
+
+ // unwrap outputs of GradCell type into Tensor type using OutputVisitor class
+ Expr tensorOutput = OutputVisitor(module_).VisitExpr(transformedExpr, transformed->ret_type);
+ return Function(f->params, tensorOutput, f->ret_type, Array<TypeVar>());
+ }
+
+ Expr VisitExpr_(const ConstantNode* op) final {
+ return CallNode::make(getGradCellConstructor(module_, "Raw"),
+ {GetRef<Constant>(op)}, Attrs(), {op->checked_type()});
+ }
+
+ Expr VisitExpr_(const CallNode* call_node) final {
+ // optimize operators
+ if (auto* op = (call_node->op).as<OpNode>()) {
+ if (op->name.compare("add") == 0 && call_node->args.size() == 2 &&
Review comment:
dont compare with name, and use op::get, with ==.
https://github.com/apache/incubator-tvm/blob/master/src/relay/backend/interpreter.cc#L431
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] MarisaKirisame commented on a change in pull
request #5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
MarisaKirisame commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r396169909
##########
File path: src/relay/transforms/gradient_cell.cc
##########
@@ -32,7 +32,29 @@
* in using the Raw constructor which means the tensor is instantiated in memory.
*
* It also overloads + and * operation which can increase performance when doing
- * operations involving zero-filled or one-filled tensors.
+ * operations involving tensors with values of only 0 or 1.
+ *
+ * Note: this pass can only be used with functions where the input/output types are
+ * a combination of TupleTypes and TensorTypes
+ *
+ * Specify optimize 6 ops:
Review comment:
```suggestion
* This pass optimizes 6 ops:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] MarisaKirisame commented on a change in pull
request #5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
MarisaKirisame commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r391387965
##########
File path: src/relay/pass/gradient.cc
##########
@@ -92,6 +92,14 @@ Expr DeGlobal(const IRModule& mod, const Expr& e) {
}
}
+std::string GradName(const Expr& e) {
Review comment:
Can you remove this function?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] hypercubestart commented on a change in pull
request #5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
hypercubestart commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r396173635
##########
File path: src/relay/transforms/gradient_cell.cc
##########
@@ -32,7 +32,29 @@
* in using the Raw constructor which means the tensor is instantiated in memory.
*
* It also overloads + and * operation which can increase performance when doing
- * operations involving zero-filled or one-filled tensors.
+ * operations involving tensors with values of only 0 or 1.
+ *
+ * Note: this pass can only be used with functions where the input/output types are
+ * a combination of TupleTypes and TensorTypes
+ *
+ * Specify optimize 6 ops:
Review comment:
done thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] jroesch commented on a change in pull request
#5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
jroesch commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r396822980
##########
File path: src/relay/transforms/gradient_cell.cc
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ *
+ * \file gradient_cell.cc
+ *
+ * \brief Convert all tensors to a Gradient Cell
+ *
+ * This pass delays or removes memory allocation by converting tensors into
+ * GradCell, an algebraic data type defined in gradient.rly
+ *
+ * This will delay or decrease memory usage. All calls to
+ * ones, ones_like, zeros, zeros_like will call the One or Zero constructor
+ * of GradCell, which will not instantiate in memory until needed. All other cases result
+ * in using the Raw constructor which means the tensor is instantiated in memory.
+ *
+ * It also overloads + and * operation which can increase performance when doing
+ * operations involving tensors with values of only 0 or 1.
+ *
+ * Note: this pass can only be used with functions where the input/output types are
+ * a combination of TupleTypes and TensorTypes
+ *
+ * This pass optimizes 6 ops:
+ * - add
+ * - multiply
+ * - ones
+ * - ones_like
+ * - zeros
+ * - zeros_like
+ *
+ * This pass makes use of three visitor. The most important one visits the entire function,
+ * one is used for wrap inputs and one to unwrap outputs.
+ *
+ * For example:
+ * fn: TensorType[(10,10), float32] -> TensorType[(10,10), float32]
+ *
+ * After this pass
+ * fn: GradCell[TensorType[(10,10), float32]] -> GradCell[TensorType[(10,10), float32]]
+ *
+ * Thus, it is necessary to wrap this outer function so that the input/output types remain the same
+ */
+
+#include <tvm/relay/analysis.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/ir/type_functor.h>
+#include <tvm/relay/transform.h>
+#include "let_list.h"
+
+namespace tvm {
+namespace relay {
+
+/*!
+* \brief Visitor to wrap inputs
+*/
+class InputVisitor: public ExprFunctor<Expr(const Expr&, const Type&)> {
+ public:
+ explicit InputVisitor(IRModule module): module_(module) {}
+
+ Expr wrapExpr(const Expr expr, const Type& type) {
+ if (type.as<TensorTypeNode>()) {
+ return Call(module_->GetConstructor("GradCell", "Raw"),
+ {expr}, Attrs(), {type});
+ } else if (auto* type_anno = type.as<TupleTypeNode>()) {
+ tvm::Array<Expr> fields;
+ for (size_t i = 0; i < type_anno->fields.size(); i++) {
+ const Type& t = type_anno->fields[i];
+ fields.push_back(this->VisitExpr(TupleGetItem(expr, i), t));
+ }
+ Expr tuple = Tuple(fields);
+ return tuple;
+ }
+
+ return expr;
+ }
+
+ Expr VisitExpr_(const VarNode* op, const Type& t) final {
+ std::cout << op->type_annotation << std::endl;
+ return wrapExpr(GetRef<Var>(op), op->type_annotation);
+ }
+
+ Expr VisitExpr_(const TupleGetItemNode* op, const Type& t) final {
+ return wrapExpr(GetRef<TupleGetItem>(op), t);
+ }
+ private:
+ IRModule module_;
+};
+
+/*!
+* \brief Visitor to unwrap output
Review comment:
Add a more descriptive comment here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] jroesch commented on a change in pull request
#5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
jroesch commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r396822885
##########
File path: src/relay/transforms/gradient_cell.cc
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ *
+ * \file gradient_cell.cc
+ *
+ * \brief Convert all tensors to a Gradient Cell
+ *
+ * This pass delays or removes memory allocation by converting tensors into
+ * GradCell, an algebraic data type defined in gradient.rly
+ *
+ * This will delay or decrease memory usage. All calls to
+ * ones, ones_like, zeros, zeros_like will call the One or Zero constructor
+ * of GradCell, which will not instantiate in memory until needed. All other cases result
+ * in using the Raw constructor which means the tensor is instantiated in memory.
+ *
+ * It also overloads + and * operation which can increase performance when doing
+ * operations involving tensors with values of only 0 or 1.
+ *
+ * Note: this pass can only be used with functions where the input/output types are
+ * a combination of TupleTypes and TensorTypes
+ *
+ * This pass optimizes 6 ops:
+ * - add
+ * - multiply
+ * - ones
+ * - ones_like
+ * - zeros
+ * - zeros_like
+ *
+ * This pass makes use of three visitor. The most important one visits the entire function,
+ * one is used for wrap inputs and one to unwrap outputs.
+ *
+ * For example:
+ * fn: TensorType[(10,10), float32] -> TensorType[(10,10), float32]
+ *
+ * After this pass
+ * fn: GradCell[TensorType[(10,10), float32]] -> GradCell[TensorType[(10,10), float32]]
+ *
+ * Thus, it is necessary to wrap this outer function so that the input/output types remain the same
+ */
+
+#include <tvm/relay/analysis.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/ir/type_functor.h>
+#include <tvm/relay/transform.h>
+#include "let_list.h"
+
+namespace tvm {
+namespace relay {
+
+/*!
+* \brief Visitor to wrap inputs
Review comment:
Add a better comment here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] hypercubestart commented on a change in pull
request #5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
hypercubestart commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r396670706
##########
File path: src/relay/transforms/gradient_cell.cc
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ *
+ * \file gradient_cell.cc
+ *
+ * \brief Convert all tensors to a Gradient Cell
+ *
+ * This pass delays or removes memory allocation by converting tensors into
+ * GradCell, an algebraic data type defined in gradient.rly
+ *
+ * This will delay or decrease memory usage. All calls to
+ * ones, ones_like, zeros, zeros_like will call the One or Zero constructor
+ * of GradCell, which will not instantiate in memory until needed. All other cases result
+ * in using the Raw constructor which means the tensor is instantiated in memory.
+ *
+ * It also overloads + and * operation which can increase performance when doing
+ * operations involving tensors with values of only 0 or 1.
+ *
+ * Note: this pass can only be used with functions where the input/output types are
+ * a combination of TupleTypes and TensorTypes
+ *
+ * This pass optimizes 6 ops:
+ * - add
+ * - multiply
+ * - ones
+ * - ones_like
+ * - zeros
+ * - zeros_like
+ *
+ * This pass makes use of three visitor. The most important one visits the entire function,
+ * one is used for wrap inputs and one to unwrap outputs.
+ *
+ * For example:
+ * fn: TensorType[(10,10), float32] -> TensorType[(10,10), float32]
+ *
+ * After this pass
+ * fn: GradCell[TensorType[(10,10), float32]] -> GradCell[TensorType[(10,10), float32]]
+ *
+ * Thus, it is necessary to wrap this outer function so that the input/output types remain the same
+ */
+
+#include <tvm/relay/analysis.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/ir/type_functor.h>
+#include <tvm/relay/transform.h>
+#include "let_list.h"
+
+namespace tvm {
+namespace relay {
+
+/*!
+* \brief Get constructor of GradCell TypeDef with name_hint
+*
+* module must have TypeDefinition of GradCell (defined in gradient.rly)
+*/
+Constructor getGradCellConstructor(IRModule module, std::string name_hint) {
+ TypeData gradCell = module->LookupTypeDef("GradCell");
Review comment:
yeah, added it as `GetConstructor` function although it may be more appropriate to be a separate PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] hypercubestart commented on issue #5039: [Relay]
GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
hypercubestart commented on issue #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#issuecomment-603437659
thanks @jroesch !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] MarisaKirisame commented on issue #5039:
GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
MarisaKirisame commented on issue #5039: GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#issuecomment-597448993
had you rebased?
you should ask some ppl to help review. since you are new I will help you this time.
@junrushao1994 @slyubomirsky @altanh @wweic can you guys help review?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] hypercubestart commented on a change in pull
request #5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
hypercubestart commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r396879116
##########
File path: src/relay/transforms/gradient_cell.cc
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ *
+ * \file gradient_cell.cc
+ *
+ * \brief Convert all tensors to a Gradient Cell
+ *
+ * This pass delays or removes memory allocation by converting tensors into
+ * GradCell, an algebraic data type defined in gradient.rly
+ *
+ * This will delay or decrease memory usage. All calls to
+ * ones, ones_like, zeros, zeros_like will call the One or Zero constructor
+ * of GradCell, which will not instantiate in memory until needed. All other cases result
+ * in using the Raw constructor which means the tensor is instantiated in memory.
+ *
+ * It also overloads + and * operation which can increase performance when doing
+ * operations involving tensors with values of only 0 or 1.
+ *
+ * Note: this pass can only be used with functions where the input/output types are
+ * a combination of TupleTypes and TensorTypes
+ *
+ * This pass optimizes 6 ops:
+ * - add
+ * - multiply
+ * - ones
+ * - ones_like
+ * - zeros
+ * - zeros_like
+ *
+ * This pass makes use of three visitor. The most important one visits the entire function,
+ * one is used for wrap inputs and one to unwrap outputs.
+ *
+ * For example:
+ * fn: TensorType[(10,10), float32] -> TensorType[(10,10), float32]
+ *
+ * After this pass
+ * fn: GradCell[TensorType[(10,10), float32]] -> GradCell[TensorType[(10,10), float32]]
+ *
+ * Thus, it is necessary to wrap this outer function so that the input/output types remain the same
+ */
+
+#include <tvm/relay/analysis.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/ir/type_functor.h>
+#include <tvm/relay/transform.h>
+#include "let_list.h"
+
+namespace tvm {
+namespace relay {
+
+/*!
+* \brief Visitor to wrap inputs
+*/
+class InputVisitor: public ExprFunctor<Expr(const Expr&, const Type&)> {
+ public:
+ explicit InputVisitor(IRModule module): module_(module) {}
+
+ Expr wrapExpr(const Expr expr, const Type& type) {
+ if (type.as<TensorTypeNode>()) {
+ return Call(module_->GetConstructor("GradCell", "Raw"),
+ {expr}, Attrs(), {type});
+ } else if (auto* type_anno = type.as<TupleTypeNode>()) {
+ tvm::Array<Expr> fields;
+ for (size_t i = 0; i < type_anno->fields.size(); i++) {
+ const Type& t = type_anno->fields[i];
+ fields.push_back(this->VisitExpr(TupleGetItem(expr, i), t));
+ }
+ Expr tuple = Tuple(fields);
+ return tuple;
+ }
+
+ return expr;
+ }
+
+ Expr VisitExpr_(const VarNode* op, const Type& t) final {
+ std::cout << op->type_annotation << std::endl;
+ return wrapExpr(GetRef<Var>(op), op->type_annotation);
+ }
+
+ Expr VisitExpr_(const TupleGetItemNode* op, const Type& t) final {
+ return wrapExpr(GetRef<TupleGetItem>(op), t);
+ }
+ private:
+ IRModule module_;
+};
+
+/*!
+* \brief Visitor to unwrap output
+*/
+class OutputVisitor: public ExprFunctor<Expr(const Expr&, const Type&)> {
+ public:
+ explicit OutputVisitor(IRModule module): module_(module) {}
+
+ Expr unwrapExpr(const Expr expr, const Type& type) {
+ if (auto* type_call = type.as<TypeCallNode>()) {
+ if (type_call->func.same_as(module_->GetGlobalTypeVar("GradCell"))) {
+ return Call(module_->GetGlobalVar("FromGradCell"), {expr});
+ }
+ return expr;
+ } else if (auto* type_anno = type.as<TupleTypeNode>()) {
+ tvm::Array<Expr> fields;
+ for (size_t i = 0; i < type_anno->fields.size(); i++) {
+ const Type& t = type_anno->fields[i];
+ fields.push_back(this->VisitExpr(TupleGetItem(expr, i), t));
+ }
+ Expr tuple = Tuple(fields);
+ return tuple;
+ }
+
+ return expr;
+ }
+
+ Expr VisitExpr_(const CallNode* op, const Type& t) final {
+ return unwrapExpr(GetRef<Call>(op), t);
+ }
+
+ Expr VisitExpr_(const TupleGetItemNode* op, const Type& t) final {
+ return unwrapExpr(GetRef<TupleGetItem>(op), t);
+ }
+ private:
+ IRModule module_;
+};
+
+class GradientCellTransform: public ExprMutator, public TypeMutator {
+ public:
+ explicit GradientCellTransform(IRModule module):
+ module_(module) {
+ module_->ImportFromStd("gradient.rly");
+ }
+
+ /*!
+ * \brief apply GradientCell transformation and wrap function
+ * so that function type stays the same
+ *
+ * input/output types should only be a combination of TupleTypes and TensorTypes
+ */
+ Expr transform(const Expr& e) {
+ auto* f = (e).as<FunctionNode>();
+ auto* transformed = this->Mutate(e).as<FunctionNode>();
+
+ if (e.same_as(GetRef<Function>(transformed))) {
+ return GetRef<Function>(transformed);
+ }
+
+ // wrap inputs of Tensor type using InputVisitor class
+ tvm::Array<Expr> args;
+ for (Var var : f->params) {
+ Expr wrappedInput = InputVisitor(module_).VisitExpr(var, var->checked_type());
+ args.push_back(wrappedInput);
+ }
+ Expr transformedExpr = Call(GetRef<Function>(transformed), args);
+
+ // unwrap outputs of GradCell type into Tensor type using OutputVisitor class
+ Expr tensorOutput = OutputVisitor(module_).VisitExpr(transformedExpr, transformed->ret_type);
+ return Function(f->params, tensorOutput, f->ret_type, Array<TypeVar>());
+ }
+
+ Expr VisitExpr_(const ConstantNode* op) final {
+ return Call(module_->GetConstructor("GradCell", "Raw"),
+ {GetRef<Constant>(op)}, Attrs(), {op->checked_type()});
+ }
+
+ Expr VisitExpr_(const CallNode* call_node) final {
+ // optimize operators
Review comment:
done, increased code readability.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] MarisaKirisame commented on a change in pull
request #5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
MarisaKirisame commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r396170061
##########
File path: src/relay/transforms/gradient_cell.cc
##########
@@ -44,19 +66,139 @@
namespace tvm {
namespace relay {
+namespace GradientCellPass { // avoid polluting namespace
Review comment:
.cc do not pollute namespace. namespace only get polluted when #include happens.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] jroesch commented on a change in pull request
#5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
jroesch commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r396823122
##########
File path: src/relay/transforms/gradient_cell.cc
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ *
+ * \file gradient_cell.cc
+ *
+ * \brief Convert all tensors to a Gradient Cell
+ *
+ * This pass delays or removes memory allocation by converting tensors into
+ * GradCell, an algebraic data type defined in gradient.rly
+ *
+ * This will delay or decrease memory usage. All calls to
+ * ones, ones_like, zeros, zeros_like will call the One or Zero constructor
+ * of GradCell, which will not instantiate in memory until needed. All other cases result
+ * in using the Raw constructor which means the tensor is instantiated in memory.
+ *
+ * It also overloads + and * operation which can increase performance when doing
+ * operations involving tensors with values of only 0 or 1.
+ *
+ * Note: this pass can only be used with functions where the input/output types are
+ * a combination of TupleTypes and TensorTypes
+ *
+ * This pass optimizes 6 ops:
+ * - add
+ * - multiply
+ * - ones
+ * - ones_like
+ * - zeros
+ * - zeros_like
+ *
+ * This pass makes use of three visitor. The most important one visits the entire function,
+ * one is used for wrap inputs and one to unwrap outputs.
+ *
+ * For example:
+ * fn: TensorType[(10,10), float32] -> TensorType[(10,10), float32]
+ *
+ * After this pass
+ * fn: GradCell[TensorType[(10,10), float32]] -> GradCell[TensorType[(10,10), float32]]
+ *
+ * Thus, it is necessary to wrap this outer function so that the input/output types remain the same
+ */
+
+#include <tvm/relay/analysis.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/ir/type_functor.h>
+#include <tvm/relay/transform.h>
+#include "let_list.h"
+
+namespace tvm {
+namespace relay {
+
+/*!
+* \brief Visitor to wrap inputs
+*/
+class InputVisitor: public ExprFunctor<Expr(const Expr&, const Type&)> {
+ public:
+ explicit InputVisitor(IRModule module): module_(module) {}
+
+ Expr wrapExpr(const Expr expr, const Type& type) {
+ if (type.as<TensorTypeNode>()) {
+ return Call(module_->GetConstructor("GradCell", "Raw"),
+ {expr}, Attrs(), {type});
+ } else if (auto* type_anno = type.as<TupleTypeNode>()) {
+ tvm::Array<Expr> fields;
+ for (size_t i = 0; i < type_anno->fields.size(); i++) {
+ const Type& t = type_anno->fields[i];
+ fields.push_back(this->VisitExpr(TupleGetItem(expr, i), t));
+ }
+ Expr tuple = Tuple(fields);
+ return tuple;
+ }
+
+ return expr;
+ }
+
+ Expr VisitExpr_(const VarNode* op, const Type& t) final {
+ std::cout << op->type_annotation << std::endl;
+ return wrapExpr(GetRef<Var>(op), op->type_annotation);
+ }
+
+ Expr VisitExpr_(const TupleGetItemNode* op, const Type& t) final {
+ return wrapExpr(GetRef<TupleGetItem>(op), t);
+ }
+ private:
+ IRModule module_;
+};
+
+/*!
+* \brief Visitor to unwrap output
+*/
+class OutputVisitor: public ExprFunctor<Expr(const Expr&, const Type&)> {
+ public:
+ explicit OutputVisitor(IRModule module): module_(module) {}
+
+ Expr unwrapExpr(const Expr expr, const Type& type) {
+ if (auto* type_call = type.as<TypeCallNode>()) {
+ if (type_call->func.same_as(module_->GetGlobalTypeVar("GradCell"))) {
+ return Call(module_->GetGlobalVar("FromGradCell"), {expr});
+ }
+ return expr;
+ } else if (auto* type_anno = type.as<TupleTypeNode>()) {
+ tvm::Array<Expr> fields;
+ for (size_t i = 0; i < type_anno->fields.size(); i++) {
+ const Type& t = type_anno->fields[i];
+ fields.push_back(this->VisitExpr(TupleGetItem(expr, i), t));
+ }
+ Expr tuple = Tuple(fields);
+ return tuple;
+ }
+
+ return expr;
+ }
+
+ Expr VisitExpr_(const CallNode* op, const Type& t) final {
+ return unwrapExpr(GetRef<Call>(op), t);
+ }
+
+ Expr VisitExpr_(const TupleGetItemNode* op, const Type& t) final {
+ return unwrapExpr(GetRef<TupleGetItem>(op), t);
+ }
+ private:
+ IRModule module_;
+};
+
+class GradientCellTransform: public ExprMutator, public TypeMutator {
+ public:
+ explicit GradientCellTransform(IRModule module):
+ module_(module) {
+ module_->ImportFromStd("gradient.rly");
+ }
+
+ /*!
+ * \brief apply GradientCell transformation and wrap function
+ * so that function type stays the same
+ *
+ * input/output types should only be a combination of TupleTypes and TensorTypes
+ */
+ Expr transform(const Expr& e) {
Review comment:
this does not match the C++ style guide
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] MarisaKirisame commented on issue #5039:
GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
MarisaKirisame commented on issue #5039: GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#issuecomment-597720628
Just to be clear, this pass save 50% memory for reverse mode ad.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] hypercubestart commented on a change in pull
request #5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
hypercubestart commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r396838571
##########
File path: src/relay/transforms/gradient_cell.cc
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ *
+ * \file gradient_cell.cc
+ *
+ * \brief Convert all tensors to a Gradient Cell
+ *
+ * This pass delays or removes memory allocation by converting tensors into
+ * GradCell, an algebraic data type defined in gradient.rly
+ *
+ * This will delay or decrease memory usage. All calls to
+ * ones, ones_like, zeros, zeros_like will call the One or Zero constructor
+ * of GradCell, which will not instantiate in memory until needed. All other cases result
+ * in using the Raw constructor which means the tensor is instantiated in memory.
+ *
+ * It also overloads + and * operation which can increase performance when doing
+ * operations involving tensors with values of only 0 or 1.
+ *
+ * Note: this pass can only be used with functions where the input/output types are
+ * a combination of TupleTypes and TensorTypes
+ *
+ * This pass optimizes 6 ops:
+ * - add
+ * - multiply
+ * - ones
+ * - ones_like
+ * - zeros
+ * - zeros_like
+ *
+ * This pass makes use of three visitor. The most important one visits the entire function,
+ * one is used for wrap inputs and one to unwrap outputs.
+ *
+ * For example:
+ * fn: TensorType[(10,10), float32] -> TensorType[(10,10), float32]
+ *
+ * After this pass
+ * fn: GradCell[TensorType[(10,10), float32]] -> GradCell[TensorType[(10,10), float32]]
+ *
+ * Thus, it is necessary to wrap this outer function so that the input/output types remain the same
+ */
+
+#include <tvm/relay/analysis.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/ir/type_functor.h>
+#include <tvm/relay/transform.h>
+#include "let_list.h"
+
+namespace tvm {
+namespace relay {
+
+/*!
+* \brief Visitor to wrap inputs
+*/
+class InputVisitor: public ExprFunctor<Expr(const Expr&, const Type&)> {
+ public:
+ explicit InputVisitor(IRModule module): module_(module) {}
+
+ Expr wrapExpr(const Expr expr, const Type& type) {
+ if (type.as<TensorTypeNode>()) {
+ return Call(module_->GetConstructor("GradCell", "Raw"),
+ {expr}, Attrs(), {type});
+ } else if (auto* type_anno = type.as<TupleTypeNode>()) {
+ tvm::Array<Expr> fields;
+ for (size_t i = 0; i < type_anno->fields.size(); i++) {
+ const Type& t = type_anno->fields[i];
+ fields.push_back(this->VisitExpr(TupleGetItem(expr, i), t));
+ }
+ Expr tuple = Tuple(fields);
+ return tuple;
+ }
+
+ return expr;
+ }
+
+ Expr VisitExpr_(const VarNode* op, const Type& t) final {
+ std::cout << op->type_annotation << std::endl;
+ return wrapExpr(GetRef<Var>(op), op->type_annotation);
+ }
+
+ Expr VisitExpr_(const TupleGetItemNode* op, const Type& t) final {
+ return wrapExpr(GetRef<TupleGetItem>(op), t);
+ }
+ private:
+ IRModule module_;
+};
+
+/*!
+* \brief Visitor to unwrap output
+*/
+class OutputVisitor: public ExprFunctor<Expr(const Expr&, const Type&)> {
+ public:
+ explicit OutputVisitor(IRModule module): module_(module) {}
+
+ Expr unwrapExpr(const Expr expr, const Type& type) {
+ if (auto* type_call = type.as<TypeCallNode>()) {
+ if (type_call->func.same_as(module_->GetGlobalTypeVar("GradCell"))) {
+ return Call(module_->GetGlobalVar("FromGradCell"), {expr});
+ }
+ return expr;
+ } else if (auto* type_anno = type.as<TupleTypeNode>()) {
+ tvm::Array<Expr> fields;
+ for (size_t i = 0; i < type_anno->fields.size(); i++) {
+ const Type& t = type_anno->fields[i];
+ fields.push_back(this->VisitExpr(TupleGetItem(expr, i), t));
+ }
+ Expr tuple = Tuple(fields);
+ return tuple;
+ }
+
+ return expr;
+ }
+
+ Expr VisitExpr_(const CallNode* op, const Type& t) final {
+ return unwrapExpr(GetRef<Call>(op), t);
+ }
+
+ Expr VisitExpr_(const TupleGetItemNode* op, const Type& t) final {
+ return unwrapExpr(GetRef<TupleGetItem>(op), t);
+ }
+ private:
+ IRModule module_;
+};
+
+class GradientCellTransform: public ExprMutator, public TypeMutator {
+ public:
+ explicit GradientCellTransform(IRModule module):
+ module_(module) {
+ module_->ImportFromStd("gradient.rly");
+ }
+
+ /*!
+ * \brief apply GradientCell transformation and wrap function
+ * so that function type stays the same
+ *
+ * input/output types should only be a combination of TupleTypes and TensorTypes
+ */
+ Expr transform(const Expr& e) {
Review comment:
done thanks for review!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] MarisaKirisame commented on a change in pull
request #5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
MarisaKirisame commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r391389331
##########
File path: src/ir/type_functor.cc
##########
@@ -151,6 +151,7 @@ Type TypeMutator::VisitType_(const FuncTypeNode* op) {
Array<Type> new_args = MutateArray(op->arg_types);
changed = changed || !new_args.same_as(op->arg_types);
+ CHECK(new_args.size() == op->arg_types.size());
Review comment:
can you remove this? it was only for debugging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] jroesch commented on a change in pull request
#5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
jroesch commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r396823371
##########
File path: tests/python/relay/test_pass_gradient_cell.py
##########
@@ -0,0 +1,401 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay import create_executor, transform
+from tvm.relay.testing import rand, run_infer_type
+from tvm.testing import assert_allclose
+import pytest
+
+def test_tc():
+ # test typechecks
Review comment:
Improve these comments or possibly convert to doc strings.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] hypercubestart commented on a change in pull
request #5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
hypercubestart commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r396173670
##########
File path: src/relay/transforms/gradient_cell.cc
##########
@@ -44,19 +66,139 @@
namespace tvm {
namespace relay {
+namespace GradientCellPass { // avoid polluting namespace
Review comment:
done thanks for the review!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] hypercubestart commented on a change in pull
request #5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
hypercubestart commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r396652494
##########
File path: src/relay/transforms/gradient_cell.cc
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ *
+ * \file gradient_cell.cc
+ *
+ * \brief Convert all tensors to a Gradient Cell
+ *
+ * This pass delays or removes memory allocation by converting tensors into
+ * GradCell, an algebraic data type defined in gradient.rly
+ *
+ * This will delay or decrease memory usage. All calls to
+ * ones, ones_like, zeros, zeros_like will call the One or Zero constructor
+ * of GradCell, which will not instantiate in memory until needed. All other cases result
+ * in using the Raw constructor which means the tensor is instantiated in memory.
+ *
+ * It also overloads + and * operation which can increase performance when doing
+ * operations involving tensors with values of only 0 or 1.
+ *
+ * Note: this pass can only be used with functions where the input/output types are
+ * a combination of TupleTypes and TensorTypes
+ *
+ * This pass optimizes 6 ops:
+ * - add
+ * - multiply
+ * - ones
+ * - ones_like
+ * - zeros
+ * - zeros_like
+ *
+ * This pass makes use of three visitor. The most important one visits the entire function,
+ * one is used for wrap inputs and one to unwrap outputs.
+ *
+ * For example:
+ * fn: TensorType[(10,10), float32] -> TensorType[(10,10), float32]
+ *
+ * After this pass
+ * fn: GradCell[TensorType[(10,10), float32]] -> GradCell[TensorType[(10,10), float32]]
+ *
+ * Thus, it is necessary to wrap this outer function so that the input/output types remain the same
+ */
+
+#include <tvm/relay/analysis.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/ir/type_functor.h>
+#include <tvm/relay/transform.h>
+#include "let_list.h"
+
+namespace tvm {
+namespace relay {
+
+/*!
+* \brief Get constructor of GradCell TypeDef with name_hint
+*
+* module must have TypeDefinition of GradCell (defined in gradient.rly)
+*/
+Constructor getGradCellConstructor(IRModule module, std::string name_hint) {
+ TypeData gradCell = module->LookupTypeDef("GradCell");
+ for (Constructor c : gradCell->constructors) {
+ if (name_hint.compare(c->name_hint) == 0) {
+ return c;
+ }
+ }
+
+ LOG(FATAL) << "Constructor " << name_hint << "not found in GradCell typedata.";
+ throw std::runtime_error("Constructor not found in GradCell typedata");
+}
+
+/*!
+* \brief Visitor to wrap inputs
+*/
+class InputVisitor: public ExprFunctor<Expr(const Expr&, const Type&)> {
+ public:
+ explicit InputVisitor(IRModule module): module_(module) {}
+
+ Expr wrapExpr(const Expr expr, const Type& type) {
+ if (type.as<TensorTypeNode>()) {
+ return CallNode::make(getGradCellConstructor(module_, "Raw"),
+ {expr}, Attrs(), {type});
+ } else if (auto* type_anno = type.as<TupleTypeNode>()) {
+ tvm::Array<Expr> fields;
+ for (size_t i = 0; i < type_anno->fields.size(); i++) {
+ const Type& t = type_anno->fields[i];
+ fields.push_back(this->VisitExpr(TupleGetItemNode::make(expr, i), t));
+ }
+ Expr tuple = TupleNode::make(fields);
+ return tuple;
+ }
+
+ return expr;
+ }
+
+ Expr VisitExpr_(const VarNode* op, const Type& t) final {
+ std::cout << op->type_annotation << std::endl;
+ return wrapExpr(GetRef<Var>(op), op->type_annotation);
+ }
+
+ Expr VisitExpr_(const TupleGetItemNode* op, const Type& t) final {
+ return wrapExpr(GetRef<TupleGetItem>(op), t);
+ }
+ private:
+ IRModule module_;
+};
+
+/*!
+* \brief Visitor to unwrap output
+*/
+class OutputVisitor: public ExprFunctor<Expr(const Expr&, const Type&)> {
+ public:
+ explicit OutputVisitor(IRModule module): module_(module) {}
+
+ Expr unwrapExpr(const Expr expr, const Type& type) {
+ if (auto* type_call = type.as<TypeCallNode>()) {
+ if (type_call->func.same_as(module_->GetGlobalTypeVar("GradCell"))) {
+ return CallNode::make(module_->GetGlobalVar("FromGradCell"), {expr});
+ }
+ return expr;
+ } else if (auto* type_anno = type.as<TupleTypeNode>()) {
+ tvm::Array<Expr> fields;
+ for (size_t i = 0; i < type_anno->fields.size(); i++) {
+ const Type& t = type_anno->fields[i];
+ fields.push_back(this->VisitExpr(TupleGetItemNode::make(expr, i), t));
+ }
+ Expr tuple = TupleNode::make(fields);
+ return tuple;
+ }
+
+ return expr;
+ }
+
+ Expr VisitExpr_(const CallNode* op, const Type& t) final {
+ return unwrapExpr(GetRef<Call>(op), t);
+ }
+
+ Expr VisitExpr_(const TupleGetItemNode* op, const Type& t) final {
+ return unwrapExpr(GetRef<TupleGetItem>(op), t);
+ }
+ private:
+ IRModule module_;
+};
+
+class GradientCellTransform: public ExprMutator, public TypeMutator {
+ public:
+ explicit GradientCellTransform(IRModule module):
+ module_(module) {
+ module_->ImportFromStd("gradient.rly");
+ }
+
+ /*!
+ * \brief apply GradientCell transformation and wrap function
+ * so that function type stays the same
+ *
+ * input/output types should only be a combination of TupleTypes and TensorTypes
+ */
+ Expr transform(const Expr& e) {
+ auto* f = (e).as<FunctionNode>();
+ auto* transformed = this->Mutate(e).as<FunctionNode>();
+
+ if (e.same_as(GetRef<Function>(transformed))) {
+ return GetRef<Function>(transformed);
+ }
+
+ // wrap inputs of Tensor type using InputVisitor class
+ tvm::Array<Expr> args;
+ for (Var var : f->params) {
+ Expr wrappedInput = InputVisitor(module_).VisitExpr(var, var->checked_type());
+ args.push_back(wrappedInput);
+ }
+ Expr transformedExpr = CallNode::make(GetRef<Function>(transformed), args);
+
+ // unwrap outputs of GradCell type into Tensor type using OutputVisitor class
+ Expr tensorOutput = OutputVisitor(module_).VisitExpr(transformedExpr, transformed->ret_type);
+ return Function(f->params, tensorOutput, f->ret_type, Array<TypeVar>());
+ }
+
+ Expr VisitExpr_(const ConstantNode* op) final {
+ return CallNode::make(getGradCellConstructor(module_, "Raw"),
+ {GetRef<Constant>(op)}, Attrs(), {op->checked_type()});
+ }
+
+ Expr VisitExpr_(const CallNode* call_node) final {
+ // optimize operators
+ if (auto* op = (call_node->op).as<OpNode>()) {
+ if (op->name.compare("add") == 0 && call_node->args.size() == 2 &&
Review comment:
done thanks for review!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] jroesch commented on a change in pull request
#5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
jroesch commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r396823191
##########
File path: src/relay/transforms/gradient_cell.cc
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ *
+ * \file gradient_cell.cc
+ *
+ * \brief Convert all tensors to a Gradient Cell
+ *
+ * This pass delays or removes memory allocation by converting tensors into
+ * GradCell, an algebraic data type defined in gradient.rly
+ *
+ * This will delay or decrease memory usage. All calls to
+ * ones, ones_like, zeros, zeros_like will call the One or Zero constructor
+ * of GradCell, which will not instantiate in memory until needed. All other cases result
+ * in using the Raw constructor which means the tensor is instantiated in memory.
+ *
+ * It also overloads + and * operation which can increase performance when doing
+ * operations involving tensors with values of only 0 or 1.
+ *
+ * Note: this pass can only be used with functions where the input/output types are
+ * a combination of TupleTypes and TensorTypes
+ *
+ * This pass optimizes 6 ops:
+ * - add
+ * - multiply
+ * - ones
+ * - ones_like
+ * - zeros
+ * - zeros_like
+ *
+ * This pass makes use of three visitor. The most important one visits the entire function,
+ * one is used for wrap inputs and one to unwrap outputs.
+ *
+ * For example:
+ * fn: TensorType[(10,10), float32] -> TensorType[(10,10), float32]
+ *
+ * After this pass
+ * fn: GradCell[TensorType[(10,10), float32]] -> GradCell[TensorType[(10,10), float32]]
+ *
+ * Thus, it is necessary to wrap this outer function so that the input/output types remain the same
+ */
+
+#include <tvm/relay/analysis.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/ir/type_functor.h>
+#include <tvm/relay/transform.h>
+#include "let_list.h"
+
+namespace tvm {
+namespace relay {
+
+/*!
+* \brief Visitor to wrap inputs
+*/
+class InputVisitor: public ExprFunctor<Expr(const Expr&, const Type&)> {
+ public:
+ explicit InputVisitor(IRModule module): module_(module) {}
+
+ Expr wrapExpr(const Expr expr, const Type& type) {
+ if (type.as<TensorTypeNode>()) {
+ return Call(module_->GetConstructor("GradCell", "Raw"),
+ {expr}, Attrs(), {type});
+ } else if (auto* type_anno = type.as<TupleTypeNode>()) {
+ tvm::Array<Expr> fields;
+ for (size_t i = 0; i < type_anno->fields.size(); i++) {
+ const Type& t = type_anno->fields[i];
+ fields.push_back(this->VisitExpr(TupleGetItem(expr, i), t));
+ }
+ Expr tuple = Tuple(fields);
+ return tuple;
+ }
+
+ return expr;
+ }
+
+ Expr VisitExpr_(const VarNode* op, const Type& t) final {
+ std::cout << op->type_annotation << std::endl;
+ return wrapExpr(GetRef<Var>(op), op->type_annotation);
+ }
+
+ Expr VisitExpr_(const TupleGetItemNode* op, const Type& t) final {
+ return wrapExpr(GetRef<TupleGetItem>(op), t);
+ }
+ private:
+ IRModule module_;
+};
+
+/*!
+* \brief Visitor to unwrap output
+*/
+class OutputVisitor: public ExprFunctor<Expr(const Expr&, const Type&)> {
+ public:
+ explicit OutputVisitor(IRModule module): module_(module) {}
+
+ Expr unwrapExpr(const Expr expr, const Type& type) {
+ if (auto* type_call = type.as<TypeCallNode>()) {
+ if (type_call->func.same_as(module_->GetGlobalTypeVar("GradCell"))) {
+ return Call(module_->GetGlobalVar("FromGradCell"), {expr});
+ }
+ return expr;
+ } else if (auto* type_anno = type.as<TupleTypeNode>()) {
+ tvm::Array<Expr> fields;
+ for (size_t i = 0; i < type_anno->fields.size(); i++) {
+ const Type& t = type_anno->fields[i];
+ fields.push_back(this->VisitExpr(TupleGetItem(expr, i), t));
+ }
+ Expr tuple = Tuple(fields);
+ return tuple;
+ }
+
+ return expr;
+ }
+
+ Expr VisitExpr_(const CallNode* op, const Type& t) final {
+ return unwrapExpr(GetRef<Call>(op), t);
+ }
+
+ Expr VisitExpr_(const TupleGetItemNode* op, const Type& t) final {
+ return unwrapExpr(GetRef<TupleGetItem>(op), t);
+ }
+ private:
+ IRModule module_;
+};
+
+class GradientCellTransform: public ExprMutator, public TypeMutator {
+ public:
+ explicit GradientCellTransform(IRModule module):
+ module_(module) {
+ module_->ImportFromStd("gradient.rly");
+ }
+
+ /*!
+ * \brief apply GradientCell transformation and wrap function
+ * so that function type stays the same
+ *
+ * input/output types should only be a combination of TupleTypes and TensorTypes
+ */
+ Expr transform(const Expr& e) {
+ auto* f = (e).as<FunctionNode>();
+ auto* transformed = this->Mutate(e).as<FunctionNode>();
+
+ if (e.same_as(GetRef<Function>(transformed))) {
+ return GetRef<Function>(transformed);
+ }
+
+ // wrap inputs of Tensor type using InputVisitor class
+ tvm::Array<Expr> args;
+ for (Var var : f->params) {
+ Expr wrappedInput = InputVisitor(module_).VisitExpr(var, var->checked_type());
+ args.push_back(wrappedInput);
+ }
+ Expr transformedExpr = Call(GetRef<Function>(transformed), args);
+
+ // unwrap outputs of GradCell type into Tensor type using OutputVisitor class
+ Expr tensorOutput = OutputVisitor(module_).VisitExpr(transformedExpr, transformed->ret_type);
+ return Function(f->params, tensorOutput, f->ret_type, Array<TypeVar>());
+ }
+
+ Expr VisitExpr_(const ConstantNode* op) final {
+ return Call(module_->GetConstructor("GradCell", "Raw"),
+ {GetRef<Constant>(op)}, Attrs(), {op->checked_type()});
+ }
+
+ Expr VisitExpr_(const CallNode* call_node) final {
+ // optimize operators
Review comment:
Either or improve or remove.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] hypercubestart commented on a change in pull
request #5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
hypercubestart commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r396879159
##########
File path: tests/python/relay/test_pass_gradient_cell.py
##########
@@ -0,0 +1,401 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay import create_executor, transform
+from tvm.relay.testing import rand, run_infer_type
+from tvm.testing import assert_allclose
+import pytest
+
+def test_tc():
+ # test typechecks
Review comment:
done, thanks for review!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] hypercubestart commented on a change in pull
request #5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
hypercubestart commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r391397809
##########
File path: tests/python/relay/test_pass_gradient_cell.py
##########
@@ -0,0 +1,384 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay import create_executor, transform
+from tvm.relay.testing import rand, run_infer_type
+from tvm.testing import assert_allclose
+import pytest
+
+def grad_cell_type(mod, shape, dtype):
+ grad_type = mod.get_global_type_var("GradCell")
+ type_arg = relay.TensorType(shape, dtype)
+ return grad_type(type_arg)
+
+def test_add():
+ mod = tvm.IRModule()
+ mod.import_from_std("gradient.rly")
Review comment:
yep, thanks for review!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] jroesch commented on a change in pull request
#5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
jroesch commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r396822650
##########
File path: python/tvm/relay/transform/transform.py
##########
@@ -219,6 +219,18 @@ def DeadCodeElimination(inline_once=False):
"""
return _ffi_api.DeadCodeElimination(inline_once)
+def GradientCell():
Review comment:
Is gradient cell really the right name for this given it is just making 1/0s lazy.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] jwfromm commented on issue #5039: [Relay]
GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
jwfromm commented on issue #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#issuecomment-602919614
It would be great if you can add some information on why this pass is useful / needed. I understand that its important to conserve memory when computing gradients, but converting all tensors to a new type just lazily evaluate constant fills seems somewhat drastic. Can you give some numbers on the impact of this pass for typical functions?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] jroesch commented on a change in pull request
#5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
jroesch commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r396823052
##########
File path: src/relay/transforms/gradient_cell.cc
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ *
+ * \file gradient_cell.cc
+ *
+ * \brief Convert all tensors to a Gradient Cell
+ *
+ * This pass delays or removes memory allocation by converting tensors into
+ * GradCell, an algebraic data type defined in gradient.rly
+ *
+ * This will delay or decrease memory usage. All calls to
+ * ones, ones_like, zeros, zeros_like will call the One or Zero constructor
+ * of GradCell, which will not instantiate in memory until needed. All other cases result
+ * in using the Raw constructor which means the tensor is instantiated in memory.
+ *
+ * It also overloads + and * operation which can increase performance when doing
+ * operations involving tensors with values of only 0 or 1.
+ *
+ * Note: this pass can only be used with functions where the input/output types are
+ * a combination of TupleTypes and TensorTypes
+ *
+ * This pass optimizes 6 ops:
+ * - add
+ * - multiply
+ * - ones
+ * - ones_like
+ * - zeros
+ * - zeros_like
+ *
+ * This pass makes use of three visitor. The most important one visits the entire function,
+ * one is used for wrap inputs and one to unwrap outputs.
+ *
+ * For example:
+ * fn: TensorType[(10,10), float32] -> TensorType[(10,10), float32]
+ *
+ * After this pass
+ * fn: GradCell[TensorType[(10,10), float32]] -> GradCell[TensorType[(10,10), float32]]
+ *
+ * Thus, it is necessary to wrap this outer function so that the input/output types remain the same
+ */
+
+#include <tvm/relay/analysis.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/ir/type_functor.h>
+#include <tvm/relay/transform.h>
+#include "let_list.h"
+
+namespace tvm {
+namespace relay {
+
+/*!
+* \brief Visitor to wrap inputs
+*/
+class InputVisitor: public ExprFunctor<Expr(const Expr&, const Type&)> {
+ public:
+ explicit InputVisitor(IRModule module): module_(module) {}
+
+ Expr wrapExpr(const Expr expr, const Type& type) {
+ if (type.as<TensorTypeNode>()) {
+ return Call(module_->GetConstructor("GradCell", "Raw"),
+ {expr}, Attrs(), {type});
+ } else if (auto* type_anno = type.as<TupleTypeNode>()) {
+ tvm::Array<Expr> fields;
+ for (size_t i = 0; i < type_anno->fields.size(); i++) {
+ const Type& t = type_anno->fields[i];
+ fields.push_back(this->VisitExpr(TupleGetItem(expr, i), t));
+ }
+ Expr tuple = Tuple(fields);
+ return tuple;
+ }
+
+ return expr;
+ }
+
+ Expr VisitExpr_(const VarNode* op, const Type& t) final {
+ std::cout << op->type_annotation << std::endl;
+ return wrapExpr(GetRef<Var>(op), op->type_annotation);
+ }
+
+ Expr VisitExpr_(const TupleGetItemNode* op, const Type& t) final {
+ return wrapExpr(GetRef<TupleGetItem>(op), t);
+ }
+ private:
+ IRModule module_;
+};
+
+/*!
+* \brief Visitor to unwrap output
+*/
+class OutputVisitor: public ExprFunctor<Expr(const Expr&, const Type&)> {
+ public:
+ explicit OutputVisitor(IRModule module): module_(module) {}
+
+ Expr unwrapExpr(const Expr expr, const Type& type) {
Review comment:
This does not match the C++ style guide.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] jroesch commented on a change in pull request
#5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
jroesch commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r396822569
##########
File path: python/tvm/relay/transform/transform.py
##########
@@ -219,6 +219,18 @@ def DeadCodeElimination(inline_once=False):
"""
return _ffi_api.DeadCodeElimination(inline_once)
+def GradientCell():
+ """Reduces memory usage of tensors with all 0s or 1s
+
+ Parameters
+ ----------
+
+ Returns
+ -------
+ ret: tvm.relay.Pass
+ The registered pass that delays or reduces memory allocation
Review comment:
```suggestion
A pass which delays and/or reduces memory allocation, by lazily allocating 0 or one filled tensors.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] hypercubestart commented on issue #5039: [Relay]
GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
hypercubestart commented on issue #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#issuecomment-602926229
@jwfromm this pass should be used in conjunction with the gradient pass. Since tensors for reverse-mode ad are instantiated as 0-filled tensors but are actually not useful during the forward pass, this PR should reduce memory allocation by 50% during the forward pass. I don't have any data on this but mathematically this would make sense
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] hypercubestart commented on a change in pull
request #5039: [Relay] GradientCell Relay Pass
Posted by GitBox <gi...@apache.org>.
hypercubestart commented on a change in pull request #5039: [Relay] GradientCell Relay Pass
URL: https://github.com/apache/incubator-tvm/pull/5039#discussion_r396838381
##########
File path: src/relay/transforms/gradient_cell.cc
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ *
+ * \file gradient_cell.cc
+ *
+ * \brief Convert all tensors to a Gradient Cell
+ *
+ * This pass delays or removes memory allocation by converting tensors into
+ * GradCell, an algebraic data type defined in gradient.rly
+ *
+ * This will delay or decrease memory usage. All calls to
+ * ones, ones_like, zeros, zeros_like will call the One or Zero constructor
+ * of GradCell, which will not instantiate in memory until needed. All other cases result
+ * in using the Raw constructor which means the tensor is instantiated in memory.
+ *
+ * It also overloads + and * operation which can increase performance when doing
+ * operations involving tensors with values of only 0 or 1.
+ *
+ * Note: this pass can only be used with functions where the input/output types are
+ * a combination of TupleTypes and TensorTypes
+ *
+ * This pass optimizes 6 ops:
+ * - add
+ * - multiply
+ * - ones
+ * - ones_like
+ * - zeros
+ * - zeros_like
+ *
+ * This pass makes use of three visitor. The most important one visits the entire function,
+ * one is used for wrap inputs and one to unwrap outputs.
+ *
+ * For example:
+ * fn: TensorType[(10,10), float32] -> TensorType[(10,10), float32]
+ *
+ * After this pass
+ * fn: GradCell[TensorType[(10,10), float32]] -> GradCell[TensorType[(10,10), float32]]
+ *
+ * Thus, it is necessary to wrap this outer function so that the input/output types remain the same
+ */
+
+#include <tvm/relay/analysis.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/ir/type_functor.h>
+#include <tvm/relay/transform.h>
+#include "let_list.h"
+
+namespace tvm {
+namespace relay {
+
+/*!
+* \brief Visitor to wrap inputs
Review comment:
done thanks for review!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services