You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2022/05/30 14:16:17 UTC
[GitHub] [tvm] billishyahao opened a new pull request, #11508: [BYOC][DNNL] Enable layer normalization in DNNL byoc.
billishyahao opened a new pull request, #11508:
URL: https://github.com/apache/tvm/pull/11508
This patch is to enable layer normalization in DNNL BYOC by providing a out-of-box rewrite pattern for combine the operators into single relay layer normalization operators as well as its implementation in dnnl json codegen.
After applying the rewrite pattern, we will observe the following dnnl function:
```
6202 def @tvmgen_default_dnnl_main_108(%dnnl_108_i0: Tensor[(1, 784, 128), float32], Inline=1, Compiler="dnnl", global_symbol="tvmgen_default_dnnl_main_108", Primitive=1) -> Tensor[(1, 784, 128), float32] {
6203 nn.layer_norm(%dnnl_108_i0, meta[relay.Constant][56] /* ty=Tensor[(128), float32] */, meta[relay.Constant][57] /* ty=Tensor[(128), float32] */) /* ty=Tensor[(1, 784, 128), float32] */
6204 }
```
Once you enable DNNL_VERBOSE flag, more informations are shown in log file as below:
```
onednn_verbose,exec,cpu,layer_normalization,simple_layer_normalization:any,forward_inference,data_f32::blocked:abc:f0 stats_undef::undef::f0 diff_undef::undef::f0,,flags:CH,1x784x128,0.0551758
```
Thanks for contributing to TVM! Please refer to guideline https://tvm.apache.org/docs/contribute/ for useful information and tips. After the pull request is submitted, please request code reviews from [Reviewers](https://github.com/apache/incubator-tvm/blob/master/CONTRIBUTORS.md#reviewers) by @ them in the pull request thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] billishyahao commented on a diff in pull request #11508: [BYOC][DNNL] Enable layer normalization in DNNL byoc.
Posted by GitBox <gi...@apache.org>.
billishyahao commented on code in PR #11508:
URL: https://github.com/apache/tvm/pull/11508#discussion_r889061002
##########
python/tvm/relay/op/contrib/dnnl.py:
##########
@@ -526,3 +527,52 @@ def visit_call(self, call):
new_mod["main"] = SubgraphRemover(subgraphs_to_remove, mod, new_mod).visit(mod["main"])
new_mod = transform.RemoveUnusedFunctions()(new_mod)
return new_mod
+
+
+class LayerNormRewrite(DFPatternCallback):
+ '''
+ A callback to rewrite the following operators into a single layer normalization operator.
+
+ 1 %4 = mean(%3, axis=[-1], keepdims=True) /* ty=Tensor[(1, 3136, 1), float32] */;
+ 2 %5 = subtract(%3, %4) /* ty=Tensor[(1, 3136, 64), float32] */;
+ 3 %6 = cast(%5, dtype="float32") /* ty=Tensor[(1, 3136, 64), float32] */;
+ 4 %7 = power(%6, 2f /* ty=float32 */) /* ty=Tensor[(1, 3136, 64), float32] */;
+ 5 %8 = mean(%7, axis=[-1], keepdims=True) /* ty=Tensor[(1, 3136, 1), float32] */;
+ 6 %9 = add(%8, 1e-05f /* ty=float32 */) /* ty=Tensor[(1, 3136, 1), float32] */;
+ 7 %10 = sqrt(%9) /* ty=Tensor[(1, 3136, 1), float32] */;
+ 8 %11 = divide(%5, %10) /* ty=Tensor[(1, 3136, 64), float32] */;
+ 9 %12 = multiply(%11, meta[relay.Constant][2] /* ty=Tensor[(64), float32] */) /* ty=Tensor[(1, 3136, 64), float32] */;
+ 10 %13 = add(%12, meta[relay.Constant][3] /* ty=Tensor[(64), float32] */) /* ty=Tensor[(1, 3136, 64), float32] */;
+ '''
+
+ def __init__(self):
+ super(LayerNormRewrite, self).__init__()
+ self.data = wildcard()
+ self.eps = wildcard()
+ self.gamma = wildcard()
+ self.beta = wildcard()
+ mu = is_op("mean")(self.data)
+ diff = is_op("subtract")(self.data, mu)
+ cdiff = diff | is_op("cast")(diff)
+ p1 = is_op("power")(cdiff, is_constant())
Review Comment:
Thanks for pointing it out. I have resolved it by specifying the const magic number in my latest revision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] billishyahao commented on pull request #11508: [BYOC][DNNL] Enable layer normalization in DNNL byoc.
Posted by GitBox <gi...@apache.org>.
billishyahao commented on PR #11508:
URL: https://github.com/apache/tvm/pull/11508#issuecomment-1149819408
> You need to resolve the conflict in `test_dnnl.py`, but since it will be modified in #11513, let's merge #11513 first.
Hi @masahi , Thanks for pointing out this. I have resolved the conflict in test_dnnl.py.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] crazydemo commented on pull request #11508: [BYOC][DNNL] Enable layer normalization in DNNL byoc.
Posted by GitBox <gi...@apache.org>.
crazydemo commented on PR #11508:
URL: https://github.com/apache/tvm/pull/11508#issuecomment-1141883470
Thanks for your contribution for BYOC-DNNL. And my suggestions are listed below:
1. I wonder if we can get better performance via running `layernorm` on `dnnl codegen` than running consecutive ops on `native codegen`. Could you please provide some performance numbers?
2. `Lint` has failed. Please run `task_lint.sh` to check the code style.
3. `UT` is required. You can add your test cases in `tests/python/contrib/test_dnnl.py` to ensure the functionality of the enabled ops.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] billishyahao commented on a diff in pull request #11508: [BYOC][DNNL] Enable layer normalization in DNNL byoc.
Posted by GitBox <gi...@apache.org>.
billishyahao commented on code in PR #11508:
URL: https://github.com/apache/tvm/pull/11508#discussion_r889061611
##########
src/runtime/contrib/dnnl/dnnl_json_runtime.cc:
##########
@@ -1084,6 +1086,47 @@ class DNNLJSONRuntime : public JSONRuntimeBase {
{DNNL_ARG_VARIANCE, variance_memory}});
}
+ void LayerNorm(const size_t& nid) {
+ auto node = nodes_[nid];
+
+ auto data_entry = node.GetInputs()[0];
+ auto gamma_entry = node.GetInputs()[1];
+ auto beta_entry = node.GetInputs()[2];
+
+ dnnl::memory::dims data_shape = nodes_[data_entry.id_].GetOpShape()[data_entry.index_];
+
+ float epsilon = std::stof(node.GetAttr<std::vector<std::string>>("epsilon")[0]);
+
+ // Memory description.
+ dnnl::memory::desc data_md = GenDNNLMemDescByShape(data_shape, dt::f32);
+
+ // LN description.
+ auto lnorm_desc = dnnl::layer_normalization_forward::desc(
+ dnnl::prop_kind::forward_inference, data_md, epsilon,
+ dnnl::normalization_flags::use_scale | dnnl::normalization_flags::use_shift);
+
+ auto lnorm_prim_desc = dnnl::layer_normalization_forward::primitive_desc(lnorm_desc, engine_);
+ auto lnorm_prim = dnnl::layer_normalization_forward(lnorm_prim_desc);
+
+ net_.push_back(lnorm_prim);
+
+ // Memories.
+ auto data_memory = BindDNNLMemory(data_entry, data_md);
+ JSONGraphNodeEntry out_entry(nid, 0);
+ auto dst_memory = BindDNNLMemory(out_entry, data_md);
+ auto scale_memory = BindDNNLMemory(gamma_entry, data_md);
+ auto shift_memory = BindDNNLMemory(beta_entry, data_md);
+ auto mean_memory = dnnl::memory(lnorm_prim_desc.mean_desc(), engine_);
+ auto variance_memory = dnnl::memory(lnorm_prim_desc.variance_desc(), engine_);
+
+ net_args_.push_back({{DNNL_ARG_SRC, data_memory},
+ {DNNL_ARG_MEAN, mean_memory},
+ {DNNL_ARG_VARIANCE, variance_memory},
+ {DNNL_ARG_SCALE, scale_memory},
+ {DNNL_ARG_SHIFT, shift_memory},
Review Comment:
Fix this issue to make it compatible to previous version of onednn. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] billishyahao commented on pull request #11508: [BYOC][DNNL] Enable layer normalization in DNNL byoc.
Posted by GitBox <gi...@apache.org>.
billishyahao commented on PR #11508:
URL: https://github.com/apache/tvm/pull/11508#issuecomment-1149606089
Hi @comaniac , @trevor-m , @mbaret , Could you take a look at this pr? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi merged pull request #11508: [BYOC][DNNL] Enable layer normalization in DNNL byoc.
Posted by GitBox <gi...@apache.org>.
masahi merged PR #11508:
URL: https://github.com/apache/tvm/pull/11508
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] billishyahao commented on a diff in pull request #11508: [BYOC][DNNL] Enable layer normalization in DNNL byoc.
Posted by GitBox <gi...@apache.org>.
billishyahao commented on code in PR #11508:
URL: https://github.com/apache/tvm/pull/11508#discussion_r889062886
##########
src/runtime/contrib/dnnl/dnnl_json_runtime.cc:
##########
@@ -1084,6 +1086,47 @@ class DNNLJSONRuntime : public JSONRuntimeBase {
{DNNL_ARG_VARIANCE, variance_memory}});
}
+ void LayerNorm(const size_t& nid) {
+ auto node = nodes_[nid];
+
+ auto data_entry = node.GetInputs()[0];
+ auto gamma_entry = node.GetInputs()[1];
+ auto beta_entry = node.GetInputs()[2];
+
+ dnnl::memory::dims data_shape = nodes_[data_entry.id_].GetOpShape()[data_entry.index_];
+
+ float epsilon = std::stof(node.GetAttr<std::vector<std::string>>("epsilon")[0]);
+
+ // Memory description.
+ dnnl::memory::desc data_md = GenDNNLMemDescByShape(data_shape, dt::f32);
Review Comment:
The issue is gone after merging with latest main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] billishyahao commented on pull request #11508: [BYOC][DNNL] Enable layer normalization in DNNL byoc.
Posted by GitBox <gi...@apache.org>.
billishyahao commented on PR #11508:
URL: https://github.com/apache/tvm/pull/11508#issuecomment-1147442715
Hi @masahi Please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] apeskov commented on a diff in pull request #11508: [BYOC][DNNL] Enable layer normalization in DNNL byoc.
Posted by GitBox <gi...@apache.org>.
apeskov commented on code in PR #11508:
URL: https://github.com/apache/tvm/pull/11508#discussion_r886041844
##########
src/runtime/contrib/dnnl/dnnl_json_runtime.cc:
##########
@@ -1084,6 +1086,47 @@ class DNNLJSONRuntime : public JSONRuntimeBase {
{DNNL_ARG_VARIANCE, variance_memory}});
}
+ void LayerNorm(const size_t& nid) {
+ auto node = nodes_[nid];
+
+ auto data_entry = node.GetInputs()[0];
+ auto gamma_entry = node.GetInputs()[1];
+ auto beta_entry = node.GetInputs()[2];
+
+ dnnl::memory::dims data_shape = nodes_[data_entry.id_].GetOpShape()[data_entry.index_];
+
+ float epsilon = std::stof(node.GetAttr<std::vector<std::string>>("epsilon")[0]);
+
+ // Memory description.
+ dnnl::memory::desc data_md = GenDNNLMemDescByShape(data_shape, dt::f32);
+
+ // LN description.
+ auto lnorm_desc = dnnl::layer_normalization_forward::desc(
+ dnnl::prop_kind::forward_inference, data_md, epsilon,
+ dnnl::normalization_flags::use_scale | dnnl::normalization_flags::use_shift);
+
+ auto lnorm_prim_desc = dnnl::layer_normalization_forward::primitive_desc(lnorm_desc, engine_);
+ auto lnorm_prim = dnnl::layer_normalization_forward(lnorm_prim_desc);
+
+ net_.push_back(lnorm_prim);
+
+ // Memories.
+ auto data_memory = BindDNNLMemory(data_entry, data_md);
+ JSONGraphNodeEntry out_entry(nid, 0);
+ auto dst_memory = BindDNNLMemory(out_entry, data_md);
+ auto scale_memory = BindDNNLMemory(gamma_entry, data_md);
+ auto shift_memory = BindDNNLMemory(beta_entry, data_md);
+ auto mean_memory = dnnl::memory(lnorm_prim_desc.mean_desc(), engine_);
+ auto variance_memory = dnnl::memory(lnorm_prim_desc.variance_desc(), engine_);
+
+ net_args_.push_back({{DNNL_ARG_SRC, data_memory},
+ {DNNL_ARG_MEAN, mean_memory},
+ {DNNL_ARG_VARIANCE, variance_memory},
+ {DNNL_ARG_SCALE, scale_memory},
+ {DNNL_ARG_SHIFT, shift_memory},
Review Comment:
DNNL_ARG_SCALE and DNNL_ARG_SHIFT was introduced only in v2.3. Unconditional using of that will break compilation with pervious version of oneDNN. Are you sure that it's your goal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] billishyahao commented on pull request #11508: [BYOC][DNNL] Enable layer normalization in DNNL byoc.
Posted by GitBox <gi...@apache.org>.
billishyahao commented on PR #11508:
URL: https://github.com/apache/tvm/pull/11508#issuecomment-1146090856
Hi @apeskov , please take a look at the latest version here. Feel free to comment more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] apeskov commented on a diff in pull request #11508: [BYOC][DNNL] Enable layer normalization in DNNL byoc.
Posted by GitBox <gi...@apache.org>.
apeskov commented on code in PR #11508:
URL: https://github.com/apache/tvm/pull/11508#discussion_r886055156
##########
src/runtime/contrib/dnnl/dnnl_json_runtime.cc:
##########
@@ -1084,6 +1086,47 @@ class DNNLJSONRuntime : public JSONRuntimeBase {
{DNNL_ARG_VARIANCE, variance_memory}});
}
+ void LayerNorm(const size_t& nid) {
+ auto node = nodes_[nid];
+
+ auto data_entry = node.GetInputs()[0];
+ auto gamma_entry = node.GetInputs()[1];
+ auto beta_entry = node.GetInputs()[2];
+
+ dnnl::memory::dims data_shape = nodes_[data_entry.id_].GetOpShape()[data_entry.index_];
+
+ float epsilon = std::stof(node.GetAttr<std::vector<std::string>>("epsilon")[0]);
+
+ // Memory description.
+ dnnl::memory::desc data_md = GenDNNLMemDescByShape(data_shape, dt::f32);
Review Comment:
You assume that data type is always f32. You should verify that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] apeskov commented on a diff in pull request #11508: [BYOC][DNNL] Enable layer normalization in DNNL byoc.
Posted by GitBox <gi...@apache.org>.
apeskov commented on code in PR #11508:
URL: https://github.com/apache/tvm/pull/11508#discussion_r886036031
##########
python/tvm/relay/op/contrib/dnnl.py:
##########
@@ -526,3 +527,52 @@ def visit_call(self, call):
new_mod["main"] = SubgraphRemover(subgraphs_to_remove, mod, new_mod).visit(mod["main"])
new_mod = transform.RemoveUnusedFunctions()(new_mod)
return new_mod
+
+
+class LayerNormRewrite(DFPatternCallback):
+ '''
+ A callback to rewrite the following operators into a single layer normalization operator.
+
+ 1 %4 = mean(%3, axis=[-1], keepdims=True) /* ty=Tensor[(1, 3136, 1), float32] */;
+ 2 %5 = subtract(%3, %4) /* ty=Tensor[(1, 3136, 64), float32] */;
+ 3 %6 = cast(%5, dtype="float32") /* ty=Tensor[(1, 3136, 64), float32] */;
+ 4 %7 = power(%6, 2f /* ty=float32 */) /* ty=Tensor[(1, 3136, 64), float32] */;
+ 5 %8 = mean(%7, axis=[-1], keepdims=True) /* ty=Tensor[(1, 3136, 1), float32] */;
+ 6 %9 = add(%8, 1e-05f /* ty=float32 */) /* ty=Tensor[(1, 3136, 1), float32] */;
+ 7 %10 = sqrt(%9) /* ty=Tensor[(1, 3136, 1), float32] */;
+ 8 %11 = divide(%5, %10) /* ty=Tensor[(1, 3136, 64), float32] */;
+ 9 %12 = multiply(%11, meta[relay.Constant][2] /* ty=Tensor[(64), float32] */) /* ty=Tensor[(1, 3136, 64), float32] */;
+ 10 %13 = add(%12, meta[relay.Constant][3] /* ty=Tensor[(64), float32] */) /* ty=Tensor[(1, 3136, 64), float32] */;
+ '''
+
+ def __init__(self):
+ super(LayerNormRewrite, self).__init__()
+ self.data = wildcard()
+ self.eps = wildcard()
+ self.gamma = wildcard()
+ self.beta = wildcard()
+ mu = is_op("mean")(self.data)
+ diff = is_op("subtract")(self.data, mu)
+ cdiff = diff | is_op("cast")(diff)
+ p1 = is_op("power")(cdiff, is_constant())
Review Comment:
you assume that `is_constant()` is strongly equal 2 but you didn't check that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] apeskov commented on a diff in pull request #11508: [BYOC][DNNL] Enable layer normalization in DNNL byoc.
Posted by GitBox <gi...@apache.org>.
apeskov commented on code in PR #11508:
URL: https://github.com/apache/tvm/pull/11508#discussion_r886052480
##########
src/runtime/contrib/dnnl/dnnl_json_runtime.cc:
##########
@@ -1084,6 +1086,47 @@ class DNNLJSONRuntime : public JSONRuntimeBase {
{DNNL_ARG_VARIANCE, variance_memory}});
}
+ void LayerNorm(const size_t& nid) {
+ auto node = nodes_[nid];
+
+ auto data_entry = node.GetInputs()[0];
+ auto gamma_entry = node.GetInputs()[1];
+ auto beta_entry = node.GetInputs()[2];
+
+ dnnl::memory::dims data_shape = nodes_[data_entry.id_].GetOpShape()[data_entry.index_];
+
+ float epsilon = std::stof(node.GetAttr<std::vector<std::string>>("epsilon")[0]);
Review Comment:
original "nn.layer_norm" has not only epsilon argument. At least `axis`, `center` and `scale`. By this code you assume that they always equal `axis = -1`, `center=true` and `scale=true`.
Could you please add support of all attributes or verify their values on codegen stage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on pull request #11508: [BYOC][DNNL] Enable layer normalization in DNNL byoc.
Posted by GitBox <gi...@apache.org>.
masahi commented on PR #11508:
URL: https://github.com/apache/tvm/pull/11508#issuecomment-1149612274
You need to resolve the conflict in `test_dnnl.py`, but since it will be modified in https://github.com/apache/tvm/pull/11513, let's merge https://github.com/apache/tvm/pull/11513 first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] billishyahao commented on a diff in pull request #11508: [BYOC][DNNL] Enable layer normalization in DNNL byoc.
Posted by GitBox <gi...@apache.org>.
billishyahao commented on code in PR #11508:
URL: https://github.com/apache/tvm/pull/11508#discussion_r889062560
##########
src/runtime/contrib/dnnl/dnnl_json_runtime.cc:
##########
@@ -1084,6 +1086,47 @@ class DNNLJSONRuntime : public JSONRuntimeBase {
{DNNL_ARG_VARIANCE, variance_memory}});
}
+ void LayerNorm(const size_t& nid) {
+ auto node = nodes_[nid];
+
+ auto data_entry = node.GetInputs()[0];
+ auto gamma_entry = node.GetInputs()[1];
+ auto beta_entry = node.GetInputs()[2];
+
+ dnnl::memory::dims data_shape = nodes_[data_entry.id_].GetOpShape()[data_entry.index_];
+
+ float epsilon = std::stof(node.GetAttr<std::vector<std::string>>("epsilon")[0]);
Review Comment:
I add ICHECK to this case. And will update later for supporting all other attributes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] apeskov commented on pull request #11508: [BYOC][DNNL] Enable layer normalization in DNNL byoc.
Posted by GitBox <gi...@apache.org>.
apeskov commented on PR #11508:
URL: https://github.com/apache/tvm/pull/11508#issuecomment-1144075748
@crazydemo Answering your question about performance.
> I wonder if we can get better performance via running layernorm on dnnl codegen than running consecutive ops on native codegen. Could you please provide some performance numbers?
Yes, there is performance benefit. At least they use different memory access approach. Consecutive ops with llvm codegen will produce sequence of fused kernel like next:
* mean. One pass through src tensor
* sub. One pass through src and dst tensor
* power + mean. One pass through src
* add + sqrt + div + mul + add. One pass through src and dst.
Totally we have 6 times traversing through data tensor for TVM codegen. DNNL implement it as single kernel and do only 4 passes through memory buffers (or 3 in case of in place memory).
In case of multi core system(xeon servers and other) normalise op is memory bound. And reduction of memory access becomes more important.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org