[Apache TVM Discuss] [Questions] [TensorRT] Seems ctx.sync() does not work while using TensorRT on Jetson Xavier NX

[Beom Woo Kang via Apache TVM Discuss <no...@discuss.tvm.ai>]

Thanks for your replies!
I checked the result with the code like below and it seems the results are the same:

    #tvm_trt_compare.py

    ...

    mod, params = relay.frontend.from_pytorch(scripted_model, shape_list)

    tgt = tvm.target.cuda()
    ctx = tvm.gpu(0)

    ###Same Input
    data = np.random.uniform(-1, 1, size=input_shape).astype(dtype)


    ###GPU with TVM
    with tvm.transform.PassContext(opt_level=args.opt_level):
        g1, m1, p1 = relay.build(mod, tgt, params=params)

    module1 = graph_runtime.create(g1, m1, ctx)
    module1.set_input(**p1)
    module1.set_input("data", data)
    module1.run()
    out_tvm = module1.get_output(0).asnumpy()


    ###GPU with TensorRT
    from tvm.relay.op.contrib.tensorrt import partition_for_tensorrt
    mod, config = partition_for_tensorrt(mod, params)

    with tvm.transform.PassContext(opt_level=args.opt_level, config={'relay.ext.tensorrt.options': config}):
        g2, m2, p2 = relay.build(mod, tgt, params=params)

    module2 = graph_runtime.create(g2, m2, ctx)
    module2.set_input(**p2)
    module2.set_input("data",data)
    module2.run()
    out_trt = module2.get_output(0).asnumpy()


    if np.all( np.abs( out_tvm - out_trt ) < 1e-5):
        print("CLEAR")
    else:
        print("FAIL")



Also Ccing @trevor-m,

But what I don't understand is that applying opt_level=0~2 gives me "FAIL", while opt_level=3 gives me "CLEAR".

    $ python3 tvm_trt_compare.py --opt_level 0
    FAIL
    $ python3 tvm_trt_compare.py --opt_level 3
    CLEAR

Does TensorRT on TVM support only with opt_level=3?

Besides, I measured the time taken with the same code base:
    
    $ python3 tvm_trt_compare.py --opt_level 3
    GPU_TVM: 6.222900390625
    GPU_TRT: 6.257080078125
    CLEAR

Maybe it is because these two different runtime modules (module1 and module2) share the same context?





---
[Visit Topic](https://discuss.tvm.apache.org/t/tensorrt-seems-ctx-sync-does-not-work-while-using-tensorrt-on-jetson-xavier-nx/9579/5) to respond.

You are receiving this because you enabled mailing list mode.

To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/9571318365e8521b499caaf3a9fe22d9ee766dfdbd3732a80af71b1c159fb38a).

[Apache TVM Discuss] [Questions] [TensorRT] Seems ctx.sync() does not work while using TensorRT on Jetson Xavier NX

[Beom Woo Kang via Apache TVM Discuss <no...@discuss.tvm.ai>]

Thanks for the tip! 

After adding the line you mentioned, the result gives me "CLEAR" even when I ran with opt_level=0.

It was very helpful:)





---
[Visit Topic](https://discuss.tvm.apache.org/t/tensorrt-seems-ctx-sync-does-not-work-while-using-tensorrt-on-jetson-xavier-nx/9579/7) to respond.

You are receiving this because you enabled mailing list mode.

To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/e49b87d484ee53fab7ff3adf64444941430c8c056d509fb6856929f46f98cd38).

[Apache TVM Discuss] [Questions] [TensorRT] Seems ctx.sync() does not work while using TensorRT on Jetson Xavier NX

["Cody H. Yu via Apache TVM Discuss" <no...@discuss.tvm.ai>]

Not sure about why opt_level=0,1,2 results in different outputs. Maybe TensorRT assumes an opt_level 3 pass is always on so the correcntess without that pass is not guaranteed, but this is just my guess.

For the performance, although again I'm not sure if this is the reason, you can add this line before each  build. This cleans the compile engine cache to avoid incorrect measurement.

```python
relay.backend.compile_engine.get().clear()
with tvm.transform.PassContext(...):
    g2, m2, p2 = relay.build(mod, tgt, params=params)
```





---
[Visit Topic](https://discuss.tvm.apache.org/t/tensorrt-seems-ctx-sync-does-not-work-while-using-tensorrt-on-jetson-xavier-nx/9579/6) to respond.

You are receiving this because you enabled mailing list mode.

To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/f9b941a08f655daabce08c74e1baf353b611a832d2cce3300ddff45ed76c8f09).