You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2020/01/22 01:00:07 UTC
[GitHub] [incubator-tvm] tqchen opened a new issue #4764: [CI][TEST] Fix
test_forward_rnn_layer for mxnet==1.5.1
tqchen opened a new issue #4764: [CI][TEST] Fix test_forward_rnn_layer for mxnet==1.5.1
URL: https://github.com/apache/incubator-tvm/issues/4764
This failure occurs when running the test on the latest CI image with mxnet-mkl==1.5.1,
This was one of blockers blocker for upgrading the ci image, see
http://ci.tvm.ai:8080/job/temp-ci-docker-staging/job/ci-stage/14/execution/node/288/log/
We will need to fix the test before upgrading the CI image. the docker image can be downloaded via `tvmai/ci-gpu:v0.57-temp`
cc @masahi @shoubhik
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] tqchen commented on issue #4764: [CI] ci-gpu update
blockers
Posted by GitBox <gi...@apache.org>.
tqchen commented on issue #4764: [CI] ci-gpu update blockers
URL: https://github.com/apache/incubator-tvm/issues/4764#issuecomment-580558384
given that the mkl part poses accuracy problem, i feel it might be a bad idea to rely on it for testing QNN(see also comment about intel dependency). would be great if we can explore generic alternatives that can test QNN. For the parser part, I think we can start by directly checking alpha equivalence of the graph as well as potentially the comparison to a simulated FP32 version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] shoubhik commented on issue #4764: [CI] ci-gpu
update blockers
Posted by GitBox <gi...@apache.org>.
shoubhik commented on issue #4764: [CI] ci-gpu update blockers
URL: https://github.com/apache/incubator-tvm/issues/4764#issuecomment-580066405
@tqchen, in that case, can we create a new docker instance specifically for testing the qnn parser?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] jikechao removed a comment on issue #4764: [CI]
ci-gpu update blockers
Posted by GitBox <gi...@apache.org>.
jikechao removed a comment on issue #4764: [CI] ci-gpu update blockers
URL: https://github.com/apache/incubator-tvm/issues/4764#issuecomment-583411064
got it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] shoubhik commented on issue #4764: [CI] ci-gpu
update blockers
Posted by GitBox <gi...@apache.org>.
shoubhik commented on issue #4764: [CI] ci-gpu update blockers
URL: https://github.com/apache/incubator-tvm/issues/4764#issuecomment-579609136
@tqchen and @icemelon9 in the meanwhile can we have this workaround to unblock the testing of mxnet qnn parser logic
```
for mode in ["rnn", "gru", "lstm"]:
verify(mode, 1, 64, 64, 1)
verify(mode, 10, 64, 64, 2)
# The following two codeblocks need to be fixed for mxnet 1.5.1-mkldnn
# verify(mode, 10, 64, 32, 2)
# verify(mode, 10, 64, 32, 2, batch=2)
verify(mode, 10, 32, 64, 1, bidirectional=True)
# The following two codeblocks need to be fixed for mxnet 1.5
# verify(mode, 10, 64, 64, 3, init_states=False)
# verify(mode, 10, 64, 64, 3, batch=2, bidirectional=True, init_states=False)
``
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] icemelon9 commented on issue #4764: [CI] ci-gpu
update blockers
Posted by GitBox <gi...@apache.org>.
icemelon9 commented on issue #4764: [CI] ci-gpu update blockers
URL: https://github.com/apache/incubator-tvm/issues/4764#issuecomment-580592009
One work around is that we can use the latest build of mxnet-mkl, which has fixed the problem.
```
pip install https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-30/dist/mxnet_mkl-1.6.0b20200130-py2.py3-none-manylinux1_x86_64.whl
```
See https://github.com/apache/incubator-mxnet/issues/17479
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] shoubhik edited a comment on issue #4764: [CI]
ci-gpu update blockers
Posted by GitBox <gi...@apache.org>.
shoubhik edited a comment on issue #4764: [CI] ci-gpu update blockers
URL: https://github.com/apache/incubator-tvm/issues/4764#issuecomment-579601888
A question for debugging this, it seems we need to run this on a g4 instance - http://ci.tvm.ai:8080/label/GPU/. What all software I need to install on g4 to run this script? Or can i log into one of the machines in Jenkins CI build to debug this issue?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] tqchen commented on issue #4764: [CI] ci-gpu update
blockers
Posted by GitBox <gi...@apache.org>.
tqchen commented on issue #4764: [CI] ci-gpu update blockers
URL: https://github.com/apache/incubator-tvm/issues/4764#issuecomment-576960374
cc @icemelon9 @kevinthesun
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] tqchen commented on issue #4764: [CI] ci-gpu update
blockers
Posted by GitBox <gi...@apache.org>.
tqchen commented on issue #4764: [CI] ci-gpu update blockers
URL: https://github.com/apache/incubator-tvm/issues/4764#issuecomment-581628702
OK, seems the conclusion so far is not to use the MKL DNN version. If we confirm that is the case, I we can go ahead and just update the pytorch and onnx rt cc @masahi can you reflect your PR to do that?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] shoubhik commented on issue #4764: [CI] ci-gpu
update blockers
Posted by GitBox <gi...@apache.org>.
shoubhik commented on issue #4764: [CI] ci-gpu update blockers
URL: https://github.com/apache/incubator-tvm/issues/4764#issuecomment-577277275
Might be a little busy today. But will surely get to it tomorrow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] shoubhik commented on issue #4764: [CI] ci-gpu
update blockers
Posted by GitBox <gi...@apache.org>.
shoubhik commented on issue #4764: [CI] ci-gpu update blockers
URL: https://github.com/apache/incubator-tvm/issues/4764#issuecomment-579601888
A question for debugging this, it seems we need to run this on a g4 instance - http://ci.tvm.ai:8080/label/GPU/. What all software I need to install on g4 to run this script?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] icemelon9 commented on issue #4764: [CI] ci-gpu
update blockers
Posted by GitBox <gi...@apache.org>.
icemelon9 commented on issue #4764: [CI] ci-gpu update blockers
URL: https://github.com/apache/incubator-tvm/issues/4764#issuecomment-577450537
I'll take a look at the MXNet test error.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] icemelon9 commented on issue #4764: [CI] ci-gpu
update blockers
Posted by GitBox <gi...@apache.org>.
icemelon9 commented on issue #4764: [CI] ci-gpu update blockers
URL: https://github.com/apache/incubator-tvm/issues/4764#issuecomment-580047528
@shoubhik @tqchen
I found out that mxnet-mkl RNN layer has some bug. If we use mxnet, the rnn layer test can pass. But if we install mxnet-mkl, even for 1.5.0, the mxnet result is wrong.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] shoubhik commented on issue #4764: [CI] ci-gpu
update blockers
Posted by GitBox <gi...@apache.org>.
shoubhik commented on issue #4764: [CI] ci-gpu update blockers
URL: https://github.com/apache/incubator-tvm/issues/4764#issuecomment-577277627
Might be a littlebusy today, will defenitely look at it tomorrow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] jikechao commented on issue #4764: [CI] ci-gpu
update blockers
Posted by GitBox <gi...@apache.org>.
jikechao commented on issue #4764: [CI] ci-gpu update blockers
URL: https://github.com/apache/incubator-tvm/issues/4764#issuecomment-583411064
got it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] tqchen closed issue #4764: [CI] ci-gpu update
blockers
Posted by GitBox <gi...@apache.org>.
tqchen closed issue #4764: [CI] ci-gpu update blockers
URL: https://github.com/apache/incubator-tvm/issues/4764
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] shoubhik commented on issue #4764: [CI] ci-gpu
update blockers
Posted by GitBox <gi...@apache.org>.
shoubhik commented on issue #4764: [CI] ci-gpu update blockers
URL: https://github.com/apache/incubator-tvm/issues/4764#issuecomment-581189877
> given that the mkl part poses accuracy problem, i feel it might be a bad idea to rely on it for testing QNN(see also comment about intel dependency). would be great if we can explore generic alternatives that can test QNN. For the parser part, I think we can start by directly checking alpha equivalence of the graph as well as potentially the comparison to a simulated FP32 version.
+ @anijain2305
@tqchen most of the code for mxnet qnn has been tested for MKLDNN. Some of the formulas used for quantization, dequantization and convolutions used my the reference Mxnet implementations(fake quantization) is not fully and thorougly tested. Also some of the optimizations available in MKLDNN are not implemented in the stock implementaion. So, my suggestion to unblock this PR would be to use the simple feed forward network test case, where we test only the graph generated by QNN. Once we have the fix from MKLDNN as well we can add proper test cases with MKLDNN at that time.
@tqchen and @icemelon9 what do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services