You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2021/03/02 04:32:04 UTC
[GitHub] [tvm] guyzsarun opened a new issue #7563: Accuracy drop when use batch and opt_level=3
guyzsarun opened a new issue #7563:
URL: https://github.com/apache/tvm/issues/7563
When autotuning with batch_size more than 1 and opt_level=3 with mxnet model multiple outputs, I encountered some accuracy drop in some of the output.
opt_level 0, 1 and 2 seems to be fine
**Environment**
TVM : 0.7.0
Mxnet: 1.6.0
OS: Ubuntu 18.04.4
**Code to reproduce**
``` python
import mxnet as mx
import tvm
import tvm.relay as relay
import numpy as np
from numpy import dot
from numpy.linalg import norm
from tvm.contrib import graph_runtime
model_name="model"
opt_level=3
input_shape = [2, 3, 160, 160]
sym, arg_params, aux_params = mx.model.load_checkpoint(model_name, 0)
input_dict={"data":input_shape}
mod, param = relay.frontend.from_mxnet(
sym, input_dict, arg_params=arg_params, aux_params=aux_params
)
ctx = tvm.cpu()
with tvm.transform.PassContext(opt_level=opt_level):
graph, lib, params = relay.build(mod, target='llvm', params=param,)
tvm_json_path=model_name+"_tvm.json"
tvm_lib_path=model_name+"_tvm.so"
tvm_params_path=model_name+"_tvm.params"
with open(tvm_params_path, "wb") as fo:
fo.write(relay.save_param_dict(params))
with open(tvm_json_path, "w") as fo:
fo.write(graph)
lib.export_library(tvm_lib_path)
```
**Comparing Accuracy**
``` python
graph = open(tvm_json_path).read()
lib = tvm.runtime.load_module(tvm_lib_path)
params = bytearray(open(tvm_params_path, "rb").read())
data = np.random.rand(2,3,160,160)
ctx = tvm.cpu(0)
module = graph_runtime.create(graph, lib, ctx)
module.load_params(params)
module.set_input('data', data)
module.run()
output = []
for i in range(module.get_num_outputs()):
prediction = module.get_output(i).asnumpy()
output.append(prediction.flatten())
ctx = mx.cpu()
sym, arg_params, aux_params = mx.model.load_checkpoint(model_name, 0)
arg_params["data"] = mx.nd.array(data.reshape(input_shape))
exe = sym.bind(ctx=ctx, args=arg_params, aux_states=aux_params, grad_req="null")
exe.forward()
output_mx = []
for pred in exe.outputs:
output_mx.append(pred.asnumpy().flatten())
for i in range(len(output_mx)):
cosim = dot(output[i], output_mx[i]) / (
norm(output[i]) * norm(output_mx[i])
)
print("Similarity Score for output {0} : {1:.2f} ".format(i + 1, cosim))
```
Results
```
Similarity Score for output 1 : 1.00
Similarity Score for output 2 : 1.00
Similarity Score for output 3 : 0.72
Similarity Score for output 4 : 1.00
Similarity Score for output 5 : 1.00
Similarity Score for output 6 : 0.59
Similarity Score for output 7 : 0.99
Similarity Score for output 8 : 1.00
Similarity Score for output 9 : 0.85
```
Link to [tvm discuss](https://discuss.tvm.apache.org/t/bug-performance-drop-with-batch-and-opt-level-3/9193)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] FrozenGene commented on issue #7563: Accuracy drop when use batch and opt_level=3
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on issue #7563:
URL: https://github.com/apache/tvm/issues/7563#issuecomment-796780183
> > @guyzsarun There is no mxnet model attachment. So I can not reproduce your issue.
>
> @FrozenGene You can use [this model](https://drive.google.com/drive/folders/1yGXAKebez7R_ZMjWWxaSBD5ixB68GUyR?usp=sharing) as already reported in [TVM discuss](https://discuss.tvm.apache.org/t/bug-performance-drop-with-batch-and-opt-level-3/9193/2).
I could reproduce this issue use the model mentioned. Besides `CanonicalizeOps`, if we just disable `AlterOpLayout` we could pass it too. If we set the target be `cuda`, we could pass too. The difference I could think of is when we set target be `llvm` and turn on ``AlterOpLayout`, the layout is `NCHWc`. Maybe we trigger some bug of `NCHWc` layout when batch size is not 1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] FrozenGene commented on issue #7563: Accuracy drop when use batch and opt_level=3
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on issue #7563:
URL: https://github.com/apache/tvm/issues/7563#issuecomment-800797312
Seems we could pass it when we disable more axis to fuse (https://discuss.tvm.apache.org/t/after-tuning-the-model-has-a-inconsistent-prediction-result-with-keras/9383/8?u=frozengene), but I don't know why we could pass it when we do this. Same issue: https://discuss.tvm.apache.org/t/after-tuning-the-model-has-a-inconsistent-prediction-result-with-keras/9383
@icemelon9 @kevinthesun Do you have any ideas? Have you met this issue when to use multi batch size?
cc @merrymercy @jcf94 @comaniac
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] gasgallo edited a comment on issue #7563: Accuracy drop when use batch and opt_level=3
Posted by GitBox <gi...@apache.org>.
gasgallo edited a comment on issue #7563:
URL: https://github.com/apache/tvm/issues/7563#issuecomment-795057859
> @guyzsarun There is no mxnet model attachment. So I can not reproduce your issue.
@FrozenGene You can use [this model](https://drive.google.com/drive/folders/1yGXAKebez7R_ZMjWWxaSBD5ixB68GUyR?usp=sharing) as already reported in [TVM discuss](https://discuss.tvm.apache.org/t/bug-performance-drop-with-batch-and-opt-level-3/9193/2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] FrozenGene edited a comment on issue #7563: Accuracy drop when use batch and opt_level=3
Posted by GitBox <gi...@apache.org>.
FrozenGene edited a comment on issue #7563:
URL: https://github.com/apache/tvm/issues/7563#issuecomment-796780183
> > @guyzsarun There is no mxnet model attachment. So I can not reproduce your issue.
>
> @FrozenGene You can use [this model](https://drive.google.com/drive/folders/1yGXAKebez7R_ZMjWWxaSBD5ixB68GUyR?usp=sharing) as already reported in [TVM discuss](https://discuss.tvm.apache.org/t/bug-performance-drop-with-batch-and-opt-level-3/9193/2).
I could reproduce this issue use the model mentioned. Besides `CanonicalizeOps`, if we just disable `AlterOpLayout` we could pass it too. If we set the target be `cuda`, we could pass too. The difference I could think of is when we set target be `llvm` and turn on `AlterOpLayout`, the layout is `NCHWc`. Maybe we trigger some bug of `NCHWc` layout when batch size is not 1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] zhiics edited a comment on issue #7563: Accuracy drop when use batch and opt_level=3
Posted by GitBox <gi...@apache.org>.
zhiics edited a comment on issue #7563:
URL: https://github.com/apache/tvm/issues/7563#issuecomment-791266563
all the used passes are listed here: https://github.com/apache/tvm/blob/main/src/relay/backend/build_module.cc#L255
can you try to disable each of the passes that run at opt_level 3 to see which one causes the accuracy problem?
one example could be:
```
with tvm.transform.PassContext(opt_level=3, disabled_pass={"AlterOpLayout"}):
lib = relay.build(...)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] FrozenGene commented on issue #7563: Accuracy drop when use batch and opt_level=3
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on issue #7563:
URL: https://github.com/apache/tvm/issues/7563#issuecomment-791252577
@guyzsarun There is no mxnet model attachment. So I can not reproduce your issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] zhiics edited a comment on issue #7563: Accuracy drop when use batch and opt_level=3
Posted by GitBox <gi...@apache.org>.
zhiics edited a comment on issue #7563:
URL: https://github.com/apache/tvm/issues/7563#issuecomment-791266563
Thanks for report this
all the used passes are listed here: https://github.com/apache/tvm/blob/main/src/relay/backend/build_module.cc#L255
can you try to disable each of the passes that run at opt_level 3 to see which one causes the accuracy problem?
one example could be:
```
with tvm.transform.PassContext(opt_level=3, disabled_pass={"AlterOpLayout"}):
lib = relay.build(...)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] YuhengHuang42 commented on issue #7563: Accuracy drop when use batch and opt_level=3
Posted by GitBox <gi...@apache.org>.
YuhengHuang42 commented on issue #7563:
URL: https://github.com/apache/tvm/issues/7563#issuecomment-811245830
Hi, I'm interested in this bug and did some experiments. Here are some findings:
1. If you do the FuseOps transform first on the relay graph, i.e.
```
seq = tvm.transform.Sequential(
[
transform.SimplifyInference(),
transform.FuseOps()
]
)
with tvm.transform.PassContext(opt_level=opt_level):
mod = seq(mod)
# build the model
# ...
```
Then the final result is correct.
2. If you use opt_level=4 to build the model, then the final result is also correct:
```
opt_level = 4
with tvm.transform.PassContext(opt_level=opt_level):
lib = relay.build(mod, target='llvm', params=param)
```
This seems pretty weird to me. So I disable some passes here, trying to dig deeper.
```
disabled_pass = ["CombineParallelConv2D", "CombineParallelDense", "CombineParallelBatchMatmul", "FastMath"]
opt_level = 4
with tvm.transform.PassContext(opt_level=opt_level, disabled_pass=disabled_pass):
lib = relay.build(mod, target='llvm', params=param)
```
As far as I know, these four passes are the only ones that are enabled by default. However, disable these passes doesn't influence the final result: the result is still correct.
My environment:
Build from source at 2988a08e3ff4a8956ac9b23e662374f6d8f7f4d9,
OS: macOS 10.15.7
As I'm new to TVM, I'm stuck here and can't dig deeper at present. Hope these info can help you find the root cause of the bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] gasgallo edited a comment on issue #7563: Accuracy drop when use batch and opt_level=3
Posted by GitBox <gi...@apache.org>.
gasgallo edited a comment on issue #7563:
URL: https://github.com/apache/tvm/issues/7563#issuecomment-795057859
> @guyzsarun There is no mxnet model attachment. So I can not reproduce your issue.
@FrozenGene You can use [this model](https://drive.google.com/drive/folders/1yGXAKebez7R_ZMjWWxaSBD5ixB68GUyR?usp=sharing) as already reported in https://discuss.tvm.apache.org/t/bug-performance-drop-with-batch-and-opt-level-3/9193/2?u=gasgallo.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] guyzsarun commented on issue #7563: Accuracy drop when use batch and opt_level=3
Posted by GitBox <gi...@apache.org>.
guyzsarun commented on issue #7563:
URL: https://github.com/apache/tvm/issues/7563#issuecomment-791326547
Disable `CanonicalizeOps` seems to be it
```
with tvm.transform.PassContext(opt_level=opt_level,disabled_pass=["CanonicalizeOps"]):
lib = relay.build(...)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] zhiics commented on issue #7563: Accuracy drop when use batch and opt_level=3
Posted by GitBox <gi...@apache.org>.
zhiics commented on issue #7563:
URL: https://github.com/apache/tvm/issues/7563#issuecomment-791266563
all the used passes are listed here: https://github.com/apache/tvm/blob/main/src/relay/backend/build_module.cc#L255
can you try to disable each of the passes that run at opt_level 3 to see which one causes the accuracy problem?
one example could be:
```
with tvm.transform.PassContext(opt_level=3, disabled_pass={"AlterOpLayout"}):
lib = relay.build(...)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] DzAvril commented on issue #7563: Accuracy drop when use batch and opt_level=3
Posted by GitBox <gi...@apache.org>.
DzAvril commented on issue #7563:
URL: https://github.com/apache/tvm/issues/7563#issuecomment-842208054
@FrozenGene I have this problem too. I'm trying to tune an onnx model by autotvm and auto-schedule, the model runs on GPU mali. Tuning with autotvm passed, but failed with accuracy drop using auto-schedule. Followed the suggestion above to disable pass `CanonicalizeOps` or `AlterOpLayout` can help. By the way, the **batch size is 1** and the opt-level is 3. Hope this info can help to locate the bug. Please let me known if you need any experiments under my environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] guyzsarun commented on issue #7563: Accuracy drop when use batch and opt_level=3
Posted by GitBox <gi...@apache.org>.
guyzsarun commented on issue #7563:
URL: https://github.com/apache/tvm/issues/7563#issuecomment-791216246
Upon further inspection I have tried disabling `pass_seqs.push_back(transform::CanonicalizeOps());` in [build_module](https://github.com/apache/tvm/blob/main/src/relay/backend/build_module.cc#L308) and the results is back to normal
```
Similarity Score for output 1 : 1.00
Similarity Score for output 2 : 1.00
Similarity Score for output 3 : 1.00
Similarity Score for output 4 : 1.00
Similarity Score for output 5 : 1.00
Similarity Score for output 6 : 1.00
Similarity Score for output 7 : 1.00
Similarity Score for output 8 : 1.00
Similarity Score for output 9 : 1.00
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] DzAvril commented on issue #7563: Accuracy drop when use batch and opt_level=3
Posted by GitBox <gi...@apache.org>.
DzAvril commented on issue #7563:
URL: https://github.com/apache/tvm/issues/7563#issuecomment-842208054
@FrozenGene I have this problem too. I'm trying to tune an onnx model by autotvm and auto-schedule, the model runs on GPU mali. Tuning with autotvm passed, but failed with accuracy drop using auto-schedule. Followed the suggestion above to disable pass `CanonicalizeOps` or `AlterOpLayout` can help. By the way, the **batch size is 1** and the opt-level is 3. Hope this info can help to locate the bug. Please let me known if you need any experiments under my environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] gasgallo commented on issue #7563: Accuracy drop when use batch and opt_level=3
Posted by GitBox <gi...@apache.org>.
gasgallo commented on issue #7563:
URL: https://github.com/apache/tvm/issues/7563#issuecomment-795057859
> @guyzsarun There is no mxnet model attachment. So I can not reproduce your issue.
You can use [this model](https://drive.google.com/drive/folders/1yGXAKebez7R_ZMjWWxaSBD5ixB68GUyR?usp=sharing) as already reported in https://discuss.tvm.apache.org/t/bug-performance-drop-with-batch-and-opt-level-3/9193/2?u=gasgallo.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org