You are viewing a plain text version of this content. The canonical link for it is here.
Posted to discuss-archive@tvm.apache.org by "Cody H. Yu via TVM Discuss" <no...@discuss.tvm.ai> on 2020/04/17 06:00:51 UTC
[TVM Discuss] [Questions] Can TVM now support batched inference?
Autotvm runs twice as long as tensorflow
You can try to use batch 1 for tuning and 500 for inference. The time should be just around (batch size) * (single batch inference time). Current TVM HCHW/NHWC conv2d does not tune the batch size, but some work is ongoing.
---
[Visit Topic](https://discuss.tvm.ai/t/can-tvm-now-support-batched-inference-autotvm-runs-twice-as-long-as-tensorflow/6405/2) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.ai/email/unsubscribe/acf7ed88147645f7a6486d1512f5cea79d5d43740b4f59ebce86a3e7535ff41d).
[TVM Discuss] [Questions] Can TVM now support batched inference?
Autotvm runs twice as long as tensorflow
Posted by adobay via TVM Discuss <no...@discuss.tvm.ai>.
Thank you very much. Tonight I will try what you said. The graph tuner throwed an exception, so i only tuned each op...
---
[Visit Topic](https://discuss.tvm.ai/t/can-tvm-now-support-batched-inference-autotvm-runs-twice-as-long-as-tensorflow/6405/5) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.ai/email/unsubscribe/f19a7aaf32caf5a3036113774093dd3f67c991e9e79b587e70a7c08a597920d5).
[TVM Discuss] [Questions] Can TVM now support batched inference?
Autotvm runs twice as long as tensorflow
Posted by "Cody H. Yu via TVM Discuss" <no...@discuss.tvm.ai>.
I am not sure if graph tuner is still applicable when cBLAS is used. Maybe @kevinthesun could provide more details about it.
---
[Visit Topic](https://discuss.tvm.ai/t/can-tvm-now-support-batched-inference-autotvm-runs-twice-as-long-as-tensorflow/6405/7) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.ai/email/unsubscribe/60a8166c135ab869336875b9e076bebf65477e66022061df74be1e43220425dd).
[TVM Discuss] [Questions] Can TVM now support batched inference?
Autotvm runs twice as long as tensorflow
Posted by adobay via TVM Discuss <no...@discuss.tvm.ai>.
Thanks, @kevinthesun, @comaniac
---
[Visit Topic](https://discuss.tvm.ai/t/can-tvm-now-support-batched-inference-autotvm-runs-twice-as-long-as-tensorflow/6405/9) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.ai/email/unsubscribe/f0173e26ee57df63686a83c63886346dccd0437e4784e0df23fbbc7956b89567).
[TVM Discuss] [Questions] Can TVM now support batched inference?
Autotvm runs twice as long as tensorflow
Posted by Yao Wang via TVM Discuss <no...@discuss.tvm.ai>.
You don't need graph tuning while using cblas.
---
[Visit Topic](https://discuss.tvm.ai/t/can-tvm-now-support-batched-inference-autotvm-runs-twice-as-long-as-tensorflow/6405/8) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.ai/email/unsubscribe/9079f03b3bd3d136b0f06b9d8e3833ec20ace79c485912688e99d7d7fb97d8a7).
[TVM Discuss] [Questions] Can TVM now support batched inference?
Autotvm runs twice as long as tensorflow
Posted by adobay via TVM Discuss <no...@discuss.tvm.ai>.
Thanks @comaniac, with batch size 500 and ```llvm -mcpu=haswell -libs=cblas```, compared with tensorflow, tvm gets 2~3X performance improvement. But the graph tuner will still throw an exception.
https://github.com/apache/incubator-tvm/issues/5369
---
[Visit Topic](https://discuss.tvm.ai/t/can-tvm-now-support-batched-inference-autotvm-runs-twice-as-long-as-tensorflow/6405/6) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.ai/email/unsubscribe/49adf097b166c594d169063a66c5ea788a64e0ed4ec608024f41bf621b2f6a67).
[TVM Discuss] [Questions] Can TVM now support batched inference?
Autotvm runs twice as long as tensorflow
Posted by "Cody H. Yu via TVM Discuss" <no...@discuss.tvm.ai>.
Dense is another issue tho. In this case you have to tune the model with batch size 500. Did you try graph tuner after tuning each op? Another option is enabling cBLAS for dense ops by setting `target=llvm -lib=cblas`
---
[Visit Topic](https://discuss.tvm.ai/t/can-tvm-now-support-batched-inference-autotvm-runs-twice-as-long-as-tensorflow/6405/4) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.ai/email/unsubscribe/461c111b680734605155696cacfc7d98345aea4766eb735eecea27ee86f137a0).
[TVM Discuss] [Questions] Can TVM now support batched inference?
Autotvm runs twice as long as tensorflow
Posted by adobay via TVM Discuss <no...@discuss.tvm.ai>.
My model does not contain conv2d, the most time-consuming op is nn.dense. Do you mean using optimized history to build the relay using batch 500 and then do inference?
---
[Visit Topic](https://discuss.tvm.ai/t/can-tvm-now-support-batched-inference-autotvm-runs-twice-as-long-as-tensorflow/6405/3) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.ai/email/unsubscribe/9f5ad887107cbfbad0abda7b64520ebfd39a8df39a1bd8f986eec359694b772e).