You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by "Aleksei-grovety (via GitHub)" <gi...@apache.org> on 2023/09/19 14:06:28 UTC
[GitHub] [tvm] Aleksei-grovety opened a new pull request, #15780: [microNPU][ETHOSU] MatMul legalization support
Aleksei-grovety opened a new pull request, #15780:
URL: https://github.com/apache/tvm/pull/15780
NPU has a restriction that weights must be constant, so the matrix multiplication operation was expressed using split, elementwise multiplication, reduce sum, concatenations operations.
cc @lhutton1, @ekalda, @leandron
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [microNPU][ETHOSU] MatMul legalization support [tvm]
Posted by "Aleksei-grovety (via GitHub)" <gi...@apache.org>.
Aleksei-grovety commented on code in PR #15780:
URL: https://github.com/apache/tvm/pull/15780#discussion_r1350128638
##########
tests/python/contrib/test_ethosu/test_codegen.py:
##########
@@ -1564,6 +1564,30 @@ def fully_connected(x):
)
+@pytest.mark.parametrize("accel_type", ["ethos-u55-256", "ethos-u65-256"])
+@pytest.mark.parametrize("ifm_shape", [(1, 16)])
Review Comment:
It works because width and channels dimensions are used for calculations, test cases are added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] Aleksei-grovety commented on pull request #15780: [microNPU][ETHOSU] MatMul legalization support
Posted by "Aleksei-grovety (via GitHub)" <gi...@apache.org>.
Aleksei-grovety commented on PR #15780:
URL: https://github.com/apache/tvm/pull/15780#issuecomment-1738916460
For a single matrix multiplication operation, the performance decreases, for case 1x16@16x8 on high-Performance (HP) subsystem on NPU we get 39.793 μs, on CPU 31.163 μs, it seems that with increasing size there is a linear dependence, for case 1x256@256x32 on NPU we get 406.993 μs, on CPU 323.863 μs. But in this situation, when there is only a matmul operation, the problem is that reshape and concatenate operations are offloaded on CPU. If we change model input to 1d array and use split and reshape operations (in this case all operations will be offloaded to NPU) for case 1x16@16x8on NPU we get 34.128 μs (it's still worse than on the CPU) but for case 1x256@256x32 on NPU we get 111.208 μs (almost three times better than on the CPU). I think it's worth adding an option to enable the offloading of the matmul operation on the NPU.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [microNPU][ETHOSU] MatMul legalization support [tvm]
Posted by "ekalda (via GitHub)" <gi...@apache.org>.
ekalda merged PR #15780:
URL: https://github.com/apache/tvm/pull/15780
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [microNPU][ETHOSU] MatMul legalization support [tvm]
Posted by "Aleksei-grovety (via GitHub)" <gi...@apache.org>.
Aleksei-grovety commented on PR #15780:
URL: https://github.com/apache/tvm/pull/15780#issuecomment-1752766832
> I suppose in the case where there is only matmul on the model the reshape and concatenate are not offloaded because there is no consumer to inline them into?
This is due to 2 inputs and the preprocess_ext_io pass. Before it, all operations are offloaded to the NPU, and after that the reshape and concatenate operations are added. And these operations are not added to the composites because MergeComposite is called before that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] Aleksei-grovety commented on pull request #15780: [microNPU][ETHOSU] MatMul legalization support
Posted by "Aleksei-grovety (via GitHub)" <gi...@apache.org>.
Aleksei-grovety commented on PR #15780:
URL: https://github.com/apache/tvm/pull/15780#issuecomment-1729398639
@tvm-bot rerun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [microNPU][ETHOSU] MatMul legalization support [tvm]
Posted by "ekalda (via GitHub)" <gi...@apache.org>.
ekalda commented on code in PR #15780:
URL: https://github.com/apache/tvm/pull/15780#discussion_r1345554154
##########
tests/python/contrib/test_ethosu/test_codegen.py:
##########
@@ -1564,6 +1564,30 @@ def fully_connected(x):
)
+@pytest.mark.parametrize("accel_type", ["ethos-u55-256", "ethos-u65-256"])
+@pytest.mark.parametrize("ifm_shape", [(1, 16)])
Review Comment:
Does it work if the first dimension of IFM is not 1? If yes, we should test it, if not, we should make it explicit in the `is_valid` checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org