You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2021/03/23 23:22:24 UTC

[GitHub] [incubator-mxnet] Zha0q1 opened a new pull request #20089: [Dummy] Attempt to fix ONNX dynamic shape for TRT

Zha0q1 opened a new pull request #20089:
URL: https://github.com/apache/incubator-mxnet/pull/20089


   This PR adds an option to specify the output shapes of a dynamic onnx graph. I also fixed an issue with the Range operator (TRT Range only supports INT32). However, I was still unable to run the dynamic bert graph on TRT. I tried it with trtexec and this is the error I got:
   ```
   [03/23/2021-23:03:08] [I] [TRT] Input filename:   /bert_dy.onnx
   [03/23/2021-23:03:08] [I] [TRT] ONNX IR version:  0.0.7
   [03/23/2021-23:03:08] [I] [TRT] Opset version:    12
   [03/23/2021-23:03:08] [I] [TRT] Producer name:    
   [03/23/2021-23:03:08] [I] [TRT] Producer version: 
   [03/23/2021-23:03:08] [I] [TRT] Domain:           
   [03/23/2021-23:03:08] [I] [TRT] Model version:    0
   [03/23/2021-23:03:08] [I] [TRT] Doc string:       
   [03/23/2021-23:03:08] [I] [TRT] ----------------------------------------------------------------
   [03/23/2021-23:03:09] [W] [TRT] /home/jenkins/workspace/OSS/L0_MergeRequest/oss/parsers/onnx/onnx2trt_utils.cpp:226: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
   [03/23/2021-23:03:09] [W] [TRT] /home/jenkins/workspace/OSS/L0_MergeRequest/oss/parsers/onnx/onnx2trt_utils.cpp:254: One or more weights outside the range of INT32 was clamped
   terminate called after throwing an instance of 'std::length_error'
     what():  cannot create std::vector larger than max_size()
   Aborted (core dumped)
   ```
   
   Both the raw export and the simplified graph by onnx-simplifier had this same error. 
   
   This is how I exported bert:
   
   ```python
   dynamic_input_shapes = [('batch', 'sequence'), ('batch', 'sequence'), ('batch',)]
           dynamic_output_shapes = [('batch', 'sequence', 768), ('batch', 768)]
           input_shapes = [(batch, seq_length), (batch, seq_length), (batch,)]
           input_types = [np.float32, np.float32, np.float32]
   
           converted_model_path = mx.contrib.onnx.export_model(sym_file, params_file, input_shapes,
                                                               input_types, onnx_file,
                                                               dynamic=True,
                                                               dynamic_input_shapes=dynamic_input_shapes,
                                                               dynamic_output_shapes=dynamic_output_shapes)
   ```
   
   <img width="838" alt="Screen Shot 2021-03-23 at 4 15 56 PM" src="https://user-images.githubusercontent.com/16669457/112230965-0dd04980-8bf3-11eb-9da0-c69146e2652d.png">
   
   We can see now the input and output shapes should be correct. This export also ran fine with onnxruntime. So I suspect there might be some incompatibility issue on TRT's side? I was not able to debug the above error message so I thought maybe the Nvidia team might be able to provide some insights? @TristonC @MoisesHer 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] Zha0q1 commented on pull request #20089: [Dummy] Attempt to fix ONNX dynamic shape for TRT

Posted by GitBox <gi...@apache.org>.
Zha0q1 commented on pull request #20089:
URL: https://github.com/apache/incubator-mxnet/pull/20089#issuecomment-805355063


   ```python 
   import mxnet as mx
   import numpy as np
   import gluoncv
   import onnxruntime
   from mxnet.test_utils import assert_almost_equal
   import json
   import os
   import pytest
   def test_dynamic_shape_bert_inference_onnxruntime(tmp_path, model):
       tmp_path = str(tmp_path)
       if True:
           import gluonnlp as nlp
           dataset = 'book_corpus_wiki_en_uncased'
           ctx = mx.cpu(0)
           model, vocab = nlp.model.get_model(
               name=model,
               ctx=ctx,
               dataset_name=dataset,
               pretrained=True,
               use_pooler=True,
               use_decoder=False,
               num_layers = 3,
               hparam_allow_override = True,
               use_classifier=False)
           model.hybridize(static_alloc=True)
           batch = 5
           seq_length = 16
           # create synthetic test data
           inputs = mx.nd.random.uniform(0, 30522, shape=(batch, seq_length), dtype='float32')
           token_types = mx.nd.random.uniform(0, 2, shape=(batch, seq_length), dtype='float32')
           valid_length = mx.nd.array([seq_length] * batch, dtype='float32')
           seq_encoding, cls_encoding = model(inputs, token_types, valid_length)
           prefix = "%s/bert_dy" % tmp_path
           model.export(prefix)
           sym_file = "%s-symbol.json" % prefix
           params_file = "%s-0000.params" % prefix
           onnx_file = "%s.onnx" % prefix
           dynamic_input_shapes = [('batch', 'sequence'), ('batch', 'sequence'), ('batch',)]
           dynamic_output_shapes = [('batch', 'sequence', 768), ('batch', 768)]
           input_shapes = [(batch, seq_length), (batch, seq_length), (batch,)]
           input_types = [np.float32, np.float32, np.float32]
           #converted_model_path = mx.contrib.onnx.export_model(sym_file, params_file, input_shapes,
           #                                                    input_types, onnx_file)
           converted_model_path = mx.contrib.onnx.export_model(sym_file, params_file, input_shapes,
                                                               input_types, onnx_file,
                                                               dynamic=True,
                                                               dynamic_input_shapes=dynamic_input_shapes,
                                                               dynamic_output_shapes=dynamic_output_shapes)
           # create onnxruntime session using the generated onnx file
           ses_opt = onnxruntime.SessionOptions()
           ses_opt.log_severity_level = 3
           session = onnxruntime.InferenceSession(onnx_file, ses_opt)
           # test on a different batch size
           batch = 5
           seq_length = 16
           inputs = mx.nd.random.uniform(0, 30522, shape=(batch, seq_length), dtype='float32')
           token_types = mx.nd.random.uniform(0, 2, shape=(batch, seq_length), dtype='float32')
           valid_length = mx.nd.array([seq_length] * batch, dtype='float32')
           seq_encoding, cls_encoding = model(inputs, token_types, valid_length)
           onnx_inputs = [inputs, token_types, valid_length]
           input_dict = dict((session.get_inputs()[i].name, onnx_inputs[i].asnumpy()) for i in range(len(onnx_inputs)))
           pred_onx, cls_onx = session.run(None, input_dict)
           assert_almost_equal(seq_encoding, pred_onx)
           assert_almost_equal(cls_encoding, cls_onx)
   test_dynamic_shape_bert_inference_onnxruntime('./bert', 'bert_12_768_12')
   ```
   
   This is the whole script to export to onnx. It’s repurposed from a test case on dynamic bert with onnxruntime 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #20089: [Dummy] Attempt to fix ONNX dynamic shape for TRT

Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #20089:
URL: https://github.com/apache/incubator-mxnet/pull/20089#issuecomment-805345241


   Hey @Zha0q1 , Thanks for submitting the PR 
   All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands: 
   - To trigger all jobs: @mxnet-bot run ci [all] 
   - To trigger specific jobs: @mxnet-bot run ci [job1, job2] 
   *** 
   **CI supported jobs**: [centos-cpu, windows-cpu, miscellaneous, clang, website, unix-gpu, windows-gpu, sanity, edge, unix-cpu, centos-gpu]
   *** 
   _Note_: 
    Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin. 
   All CI tests must pass before the PR can be merged. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] Zha0q1 edited a comment on pull request #20089: [Dummy] Attempt to fix ONNX dynamic shape for TRT

Posted by GitBox <gi...@apache.org>.
Zha0q1 edited a comment on pull request #20089:
URL: https://github.com/apache/incubator-mxnet/pull/20089#issuecomment-805348590


   I should add that I am using `nvcr.io/nvidia/tensorrt:21.02-py3` and onnx version 1.7.0
   
   In the TRT docker container i just did `./trtexec --onnx=/bert_dy.onnx`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] Zha0q1 commented on pull request #20089: [Dummy] Attempt to fix ONNX dynamic shape for TRT

Posted by GitBox <gi...@apache.org>.
Zha0q1 commented on pull request #20089:
URL: https://github.com/apache/incubator-mxnet/pull/20089#issuecomment-805348590


   I should add that I am using `nvcr.io/nvidia/tensorrt:21.02-py3` and onnx version 1.7.0
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org