[GitHub] [tvm] yixunzhang opened a new issue, #14947: [Bug] Wrong Result by nn.Dense and relay.Constant with TensorRT

["yixunzhang (via GitHub)" <gi...@apache.org>]

yixunzhang opened a new issue, #14947:
URL: https://github.com/apache/tvm/issues/14947

   ### Description
   I am comparing the inference result of models with and without TensorRT as the backend.
   
   My simple PyTorch model `MatMul` was translated to operator nn.dense with an input a relay.constant, and produces an all-zero result.
   
   I referred to the test case "test_batch_matmul" in tvm:tests/python/contrib/test_tensorrt.py. it works well with `matmul`, but when I changed the operator into `dense` and a parameter to `constant`, it produces the all-zero result.  
   
   PyTorch Model 
   ```
   class MatMul:
      def __init__(self):
          super(MatMul, self).__init__()
          self.w = Parameter(data = torch.ones(1024, 1024))
       def forward(self, x):
           return torch.matmul(x, self.w)
   ```
     
   ### Expected behavior
   Same result w/wo TensorRT。
   
   ### Actual behavior
   The result produced by TensorRT is all-zero.
   
   ### Environment
   TVM: 0.11.1
   TensorRT: 8.2.0.6
   Python 3.9.1
   
   ### Steps to reproduce
   ``` python
   y_shape =(64,64)
   data_type="float32" 
   target="cuda"
   dev = tvm.device(target)
   _y = tvm.nd.array(np.random.rand(*y_shape).astype(dtype=data_type))
   y=relay.Constant(_y)
   
   def get_graph(x_shape=(12, 128, 64), y_shape=(12,128,64), transa=True, transb=True): 
       x = relay.var("x",shape=(x_shape), dtype="float32")
       out = relay.nn.dense(x, y)
       f=relay.Function([x],out)
       return f, {"x":x_shape}, []
   
   f, input_shapes, is_param = get_graph(x_shape, y_shape, transa, transb)
   params = { x : np.random.uniform(-1,1, input_shapes[x]).astype(dtype=data_type) for x in is_param } 
   input_dict ={
       k: np.random.uniform(-1,1, v).astype(dtype=data_type) 
       for k, v in input_shapes.items()
       if k not in is_param
   }
   
   result_dict = dict()
   for mode in ["vm","graph"]:
       for use_trt in [True, False]:
           mod=tvm.IRModule()
           mod["main"] = f
           result_key = mode + (" trt" if use trt else "")
           if use_trt:
               use_fp16 = data_type == "float16"
               trt_target = tvm.target.Target(f"tensorrt -use_fp16={use_fp16}")
               mod = relay.transform.InferType()(mod)
               mod = tensorrt.partition_for_tensorrt(mod, params=params,target=trt_target)
               print(mod)
               with tvm.transform.PassContext(opt_level=3):
                   func = relay.create_executor(
                       mode, mod=mod,device=dev, target=[target, trt_target]
                   ).evaluate()
           else:
               mod = relay.transform.InferType()(mod)
               with tvm.transform.PassContext(opt_level=3):
                   func =relay.create_executor(
                       mode,mod=mod,device=dev,target=target
                   ).evaluate()
           result_dict[result_key] = func(**input_dict,**params)
           print(f"{result_key}", result_dict[result_key].numpy() [:10,0])
   ```
   
   ### Triage
   * byoc:tensorrt
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org