You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2022/12/19 09:59:29 UTC
[GitHub] [tvm] Qianshui-Jiang opened a new pull request, #13642: [Tensorize][TOPI] Add support for Intel® AMX(Advanced Matrix Extensions) through Tensor intrinsics
Qianshui-Jiang opened a new pull request, #13642:
URL: https://github.com/apache/tvm/pull/13642
In this PR, we add support for Intel® AMX(Advanced Matrix Extensions) through Tensor intrinsics;
we chose to enabled the AMX support and INT8 intrinsics and dense op at first, which includes:
1. Add building options and LLVM AMX intrinsic target detection ;
2. Register a global AMX init and config function;
3. LLVM AMX intrinsic integrated as Tensor intrinsic (1 for microkernel computation and 1 for accumulation tile store);
4. Add test case on u8s8s32 matmul usding the AMX Tensor intrinsics;
5. Integration of int8 dense kernel, testcase also added;
BF16 and more op are going to be considered in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] cbalint13 commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
cbalint13 commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1058190629
##########
python/tvm/topi/x86/tensor_intrin.py:
##########
@@ -348,3 +348,227 @@ def _instr(index):
binds={data: a_buffer, kernel: b_buffer},
default_buffer_params=buffer_params,
)
+
+
+def dot_32x128x32_u8s8s32_sapphirerapids(LDA):
+ """
+ Int8 dot product by every 16x64 elements using AMX-TMUL Sapphire Rapids instructions.
+ The tdpxxd instruction takes two tile of uint8 and int8 datatype -- data[16][64] and
+ kernel[1][16][16][4] -- and computes a dot product of data[16][16] in int32 datatype.
+
+ (Physically, to efficiently leveraging the tile register, we constructing a 2x2 tiles
+ matmul which performs 32x128x32 in total)
+
+ The pseudo code is as follows:
+ for(k=0; k<2; k++){
+ for(n=0; n<2; n++){
+ tileload64(tmm_b, B)
+ for(m=0; m<2; m++){
+ if(n==0)
+ tileload64(tmm_a, A)
Review Comment:
@masahi ,
I have a question related to your proposed here:
* Is there a TIR ```*_desc()``` equivalent for TOPI's body, reset, update ?
topi intrin decl: https://github.com/apache/tvm/blob/main/python/tvm/topi/x86/tensor_intrin.py#L140
* How the ```accum``` steps (very inner) can be succeeded by ```update``` (outer) or preceded by ```reset``` ?
I am looking for any TVM reference (code, example) for this, trying even finer granularity control of tensorization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi merged pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
masahi merged PR #13642:
URL: https://github.com/apache/tvm/pull/13642
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
masahi commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1059717256
##########
python/tvm/topi/x86/dense.py:
##########
@@ -293,16 +324,13 @@ def dense_vnni_compute(cfg, X, packed_w, bias=None):
),
axis=ak,
),
- tag="dense_vnni",
- attrs={"schedule_rule": "dense_vnni"},
Review Comment:
Yes, but it is important that we specify that we use this compute for VNNI. If the schedule rule annotation only says "dense_int8", we don't know which intrinsic to tensorize this compute.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
masahi commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1059735746
##########
python/tvm/topi/x86/dense.py:
##########
@@ -293,16 +324,13 @@ def dense_vnni_compute(cfg, X, packed_w, bias=None):
),
axis=ak,
),
- tag="dense_vnni",
- attrs={"schedule_rule": "dense_vnni"},
Review Comment:
Of course the test still works, because it was already written for VNNI. The point is that the name `dense_vnni` tells that the particular TE expression is meant for VNNI tensorization, in particular the weight is pre-packed appropriately. But a generic name like `dense_int8` doesn't provide such information. Just because AMX can use the same layout doesn't mean we can use `dense_int8`. If we find a compute annotated with `dense_int8`, MetaSchedule cannot tell if it should apply VNNI or AMX tensorization (if the latter is supported by MS in the future).
So please revert that commit and restore and pass `attrs={"schedule_rule": "meta_schedule.x86.dense_vnni"}` when you create a compute expression for VNNI. `schedule_rule` is not relevant for AMX for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] cbalint13 commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
cbalint13 commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1058190629
##########
python/tvm/topi/x86/tensor_intrin.py:
##########
@@ -348,3 +348,227 @@ def _instr(index):
binds={data: a_buffer, kernel: b_buffer},
default_buffer_params=buffer_params,
)
+
+
+def dot_32x128x32_u8s8s32_sapphirerapids(LDA):
+ """
+ Int8 dot product by every 16x64 elements using AMX-TMUL Sapphire Rapids instructions.
+ The tdpxxd instruction takes two tile of uint8 and int8 datatype -- data[16][64] and
+ kernel[1][16][16][4] -- and computes a dot product of data[16][16] in int32 datatype.
+
+ (Physically, to efficiently leveraging the tile register, we constructing a 2x2 tiles
+ matmul which performs 32x128x32 in total)
+
+ The pseudo code is as follows:
+ for(k=0; k<2; k++){
+ for(n=0; n<2; n++){
+ tileload64(tmm_b, B)
+ for(m=0; m<2; m++){
+ if(n==0)
+ tileload64(tmm_a, A)
Review Comment:
@masahi ,
I have a question related to your proposed here:
* Is there a TIR ```*_desc()``` equivalent for ```TOPI``` body, reset, update ?
topi intrin decl: https://github.com/apache/tvm/blob/main/python/tvm/topi/x86/tensor_intrin.py#L140
* How (TIR) ```accum``` steps (very inner) can be succeeded by ```update``` (outer) or preceded by ```reset``` ?
I am looking for any TVM reference (code, example) for this, trying even finer granularity control of autotensorization with TIR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
masahi commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1061991180
##########
apps/microtvm/zephyr/template_project/qemu-hack/qemu-system-arm:
##########
@@ -1 +0,0 @@
-qemu-system-i386
Review Comment:
Is this change relevant? If not remove it.
##########
python/tvm/relay/op/strategy/x86.py:
##########
@@ -591,18 +591,17 @@ def dense_strategy_cpu(attrs, inputs, out_type, target):
def dense_pack_strategy_cpu(attrs, inputs, out_type, target):
"""dense_pack x86 strategy"""
strategy = _op.OpStrategy()
-
if (
inputs[0].dtype == "uint8"
and inputs[1].dtype == "int8"
and out_type.dtype == "int32"
and attrs["weight_layout"] == "NC16n4c"
):
Review Comment:
I see, that works too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] Qianshui-Jiang commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
Qianshui-Jiang commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1059747493
##########
python/tvm/topi/x86/dense.py:
##########
@@ -293,16 +324,13 @@ def dense_vnni_compute(cfg, X, packed_w, bias=None):
),
axis=ak,
),
- tag="dense_vnni",
- attrs={"schedule_rule": "dense_vnni"},
Review Comment:
@masahi yep, got it, the schedule rule for ms dense_vnni are restored.
Keep it remained but AMX not use it for now.
( A smalll question: How do we know inside of this compute expression if it's created for VNNI not AMX? )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] Qianshui-Jiang commented on pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
Qianshui-Jiang commented on PR #13642:
URL: https://github.com/apache/tvm/pull/13642#issuecomment-1365242128
@cbalint13 Thanks for your curiousity!
In our test, Compared to TVM testcase `test_fc_int8_acc32()`, AMX int8 dense performes about 8X faster in single physical core.
(Since AMX machine not officially lunched yet, I can only gives a little perf data, but it's comming soon~)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] Qianshui-Jiang commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
Qianshui-Jiang commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1058782058
##########
python/tvm/topi/x86/tensor_intrin.py:
##########
@@ -348,3 +348,227 @@ def _instr(index):
binds={data: a_buffer, kernel: b_buffer},
default_buffer_params=buffer_params,
)
+
+
+def dot_32x128x32_u8s8s32_sapphirerapids(LDA):
+ """
+ Int8 dot product by every 16x64 elements using AMX-TMUL Sapphire Rapids instructions.
+ The tdpxxd instruction takes two tile of uint8 and int8 datatype -- data[16][64] and
+ kernel[1][16][16][4] -- and computes a dot product of data[16][16] in int32 datatype.
+
+ (Physically, to efficiently leveraging the tile register, we constructing a 2x2 tiles
+ matmul which performs 32x128x32 in total)
+
+ The pseudo code is as follows:
+ for(k=0; k<2; k++){
+ for(n=0; n<2; n++){
+ tileload64(tmm_b, B)
+ for(m=0; m<2; m++){
+ if(n==0)
+ tileload64(tmm_a, A)
Review Comment:
> Have you considered tensorizing at finer granularity? Meaning, instead of tensoring the whole load + compute as one tensor intrin, tensorize each load and compute separately. That could make the hard coded outer-loop factors (2 in this intrin) tunable.
Yes we considered to make it works like tensor core which mentioned here: https://github.com/apache/tvm/issues/4052, but the problem is AMX intrinsics in LLVM need user to assigned register(tmm) number by hand craft. If we seperate the load and compute, we have to leave the register assginment to schedule method, that would bring much redundancy work. So finnally we decide to follow BRGEMM and leave the whole smallest comute unit for tensorizing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
masahi commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1059735746
##########
python/tvm/topi/x86/dense.py:
##########
@@ -293,16 +324,13 @@ def dense_vnni_compute(cfg, X, packed_w, bias=None):
),
axis=ak,
),
- tag="dense_vnni",
- attrs={"schedule_rule": "dense_vnni"},
Review Comment:
Of course the test still works, because it was already written for VNNI. The point is that the name `dense_vnni` tells that the particular TE expression is meant for VNNI tensorization, in particular the weight is pre-packed appropriately. But a generic name like `dense_int8` doesn't provide such information. Just because AMX can use the same layout doesn't mean we can use `dense_int8`. If MetaSchedule finds a compute annotated with `dense_int8`, it cannot tell if it should apply VNNI or AMX tensorization (if the latter is supported by MS in the future).
So please revert that commit and restore and pass `attrs={"schedule_rule": "meta_schedule.x86.dense_vnni"}` when you create a compute expression for VNNI. `schedule_rule` is not relevant for AMX for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] cbalint13 commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
cbalint13 commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1058190629
##########
python/tvm/topi/x86/tensor_intrin.py:
##########
@@ -348,3 +348,227 @@ def _instr(index):
binds={data: a_buffer, kernel: b_buffer},
default_buffer_params=buffer_params,
)
+
+
+def dot_32x128x32_u8s8s32_sapphirerapids(LDA):
+ """
+ Int8 dot product by every 16x64 elements using AMX-TMUL Sapphire Rapids instructions.
+ The tdpxxd instruction takes two tile of uint8 and int8 datatype -- data[16][64] and
+ kernel[1][16][16][4] -- and computes a dot product of data[16][16] in int32 datatype.
+
+ (Physically, to efficiently leveraging the tile register, we constructing a 2x2 tiles
+ matmul which performs 32x128x32 in total)
+
+ The pseudo code is as follows:
+ for(k=0; k<2; k++){
+ for(n=0; n<2; n++){
+ tileload64(tmm_b, B)
+ for(m=0; m<2; m++){
+ if(n==0)
+ tileload64(tmm_a, A)
Review Comment:
@masahi ,
I have a question related to your proposed here:
* Is there a ```TIR``` *_desc() equivalent for ```TOPI``` body, reset, update ?
topi intrin decl: https://github.com/apache/tvm/blob/main/python/tvm/topi/x86/tensor_intrin.py#L140
* How (TIR) ```accum``` steps (very inner) can be succeeded by ```update``` (outer) or preceded by ```reset``` ?
I am looking for any TVM reference (code, example) for this, trying even finer granularity control of autotensorization with TIR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] Qianshui-Jiang commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
Qianshui-Jiang commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1059723468
##########
python/tvm/topi/x86/dense.py:
##########
@@ -293,16 +324,13 @@ def dense_vnni_compute(cfg, X, packed_w, bias=None):
),
axis=ak,
),
- tag="dense_vnni",
- attrs={"schedule_rule": "dense_vnni"},
Review Comment:
> This shouldn't be removed, it is used here
>
> https://github.com/apache/tvm/blob/b16a64d6edb9fd1a014fc51995dff7d0e2f4c84e/tests/python/unittest/test_meta_schedule_vnni_integration.py#L198
>
> .
> Since this only affects MetaSchedule, you don't have to provide this value for AMX. So only when `dense_int8_compute` is called for VNNI, you need to provide this attribute.
@masahi Sorry, may given the misunderstanding, I mean that can we use the `dense_int8` in this test case? cuz here inject the VNNI intrinsic explicitly.
And by default in relay build flow, it will check if the VNNI or AMX is availiable and chose different schedulling.
I've verified that this test case still functional after the little modification [in this commit](https://github.com/apache/tvm/pull/13642/commits/5718a059c69972cf71ea082a3303b5c29fa2d21f) bellow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] Qianshui-Jiang commented on pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
Qianshui-Jiang commented on PR #13642:
URL: https://github.com/apache/tvm/pull/13642#issuecomment-1365952931
@masahi Sure, you can use SDE by assign platform info like `-spr`,
But if you are not built on 5.16 kernel, you have to comment all the system related code `runtime.amx_init` ,
and don't need to init amx before the testcase when using SDE;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] cbalint13 commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
cbalint13 commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1058846644
##########
python/tvm/topi/x86/tensor_intrin.py:
##########
@@ -348,3 +348,227 @@ def _instr(index):
binds={data: a_buffer, kernel: b_buffer},
default_buffer_params=buffer_params,
)
+
+
+def dot_32x128x32_u8s8s32_sapphirerapids(LDA):
+ """
+ Int8 dot product by every 16x64 elements using AMX-TMUL Sapphire Rapids instructions.
+ The tdpxxd instruction takes two tile of uint8 and int8 datatype -- data[16][64] and
+ kernel[1][16][16][4] -- and computes a dot product of data[16][16] in int32 datatype.
+
+ (Physically, to efficiently leveraging the tile register, we constructing a 2x2 tiles
+ matmul which performs 32x128x32 in total)
+
+ The pseudo code is as follows:
+ for(k=0; k<2; k++){
+ for(n=0; n<2; n++){
+ tileload64(tmm_b, B)
+ for(m=0; m<2; m++){
+ if(n==0)
+ tileload64(tmm_a, A)
Review Comment:
@masahi ,
@Qianshui-Jiang ,
Don't want hold up the PR, I think it is fine to merge with this startup AMX implementation.
I leave here a possible way to workaround the rigidity of LLVM's AMX related intrinsic declaration:
* Use ```tvm.tir.call_extern``` insead of ```tvm.tir.call_llvm_intrin``` like [here](https://github.com/apache/tvm/blob/8551a5c71faaa2eedc8b1c49a39be0fe4688c021/python/tvm/topi/arm_cpu/mprofile/dsp/micro_kernel/gemm.py#L107-L109) .
* You can than pass argument (e.g. register number, etc ...) into the custom called function.
* You can use the argument inside that function and emit the AMX ISA as bytes: [ __asm(.word packed())](https://github.com/cbalint13/OLIMP/blob/main/demo/icebreaker/src/rv32-custom.h#L52-L53)
This is how it is done to emit custom part of RISC-V ISA, that is also too rigid using just the LLVM declarations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tvm-bot commented on pull request #13642: [Tensorize][TOPI] Add support for Intel® AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
tvm-bot commented on PR #13642:
URL: https://github.com/apache/tvm/pull/13642#issuecomment-1357386156
<!---bot-comment-->
Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from [Reviewers](https://github.com/apache/incubator-tvm/blob/master/CONTRIBUTORS.md#reviewers) by @-ing them in a comment.
<!--bot-comment-ccs-start-->
* No users to tag found in teams: `tensorize`, `topi` <sub>See [#10317](https://github.com/apache/tvm/issues/10317) for details</sub><!--bot-comment-ccs-end-->
<sub>Generated by [tvm-bot](https://github.com/apache/tvm/blob/main/ci/README.md#github-actions)</sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] cbalint13 commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
cbalint13 commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1056877284
##########
src/runtime/contrib/amx/amx_config.cc:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*
+ * \file src/runtime/contrib/amx/amx_config.cc
+ * \brief extraction of AMX configuration on x86 platforms
+ */
+#include <tvm/runtime/packed_func.h>
+#include <tvm/runtime/registry.h>
+
+namespace tvm {
+namespace runtime {
+
+#ifdef __linux__
+#include <dmlc/logging.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <immintrin.h>
+#include <signal.h>
+#include <sys/stat.h>
+#include <sys/syscall.h>
+#include <unistd.h>
+
+#define XFEATURE_XTILECFG 17
+#define XFEATURE_XTILEDATA 18
+#define XFEATURE_MASK_XTILECFG (1 << XFEATURE_XTILECFG)
+#define XFEATURE_MASK_XTILEDATA (1 << XFEATURE_XTILEDATA)
+#define XFEATURE_MASK_XTILE (XFEATURE_MASK_XTILECFG | XFEATURE_MASK_XTILEDATA)
+#define ARCH_GET_XCOMP_PERM 0x1022
+#define ARCH_REQ_XCOMP_PERM 0x1023
+
+typedef struct __tile_config {
+ uint8_t palette_id;
+ uint8_t start_row;
+ uint8_t reserved_0[14];
+ uint16_t colsb[8]; /* Colum size of each tmm register in bytes */
+ uint16_t reserved_1[8];
+ uint8_t rows[8]; /* Row size of each tmm reg in bytes */
+ uint8_t reserved_2[8];
+} __tilecfg;
+
+typedef union __union_tile_config {
+ __tilecfg s;
+ uint8_t a[64];
+} __tilecfg_u;
+
+void init_tile_config(__tilecfg_u* dst, uint16_t cols, uint8_t rows) {
+ dst->s.palette_id = 1;
+ dst->s.start_row = 0;
+
+ for (int i = 0; i < 14; i++) dst->s.reserved_0[i] = 0;
+
+ for (int i = 0; i < 8; i++) {
+ dst->s.colsb[i] = cols;
+ dst->s.rows[i] = rows;
+ dst->s.reserved_1[i] = 0;
+ dst->s.reserved_2[i] = 0;
+ }
+
+ _tile_loadconfig(dst->a);
+}
+
+TVM_REGISTER_GLOBAL("runtime.amx_tileconfig").set_body([](TVMArgs args, TVMRetValue* rv) {
+ int rows = args[0];
+ int cols = args[1];
+ LOG(INFO) << "rows: " << rows << ", cols:" << cols;
+ // -----------Config for AMX tile resgister----------------------
Review Comment:
s|resgister|register|
##########
python/tvm/topi/x86/dense_alter_op.py:
##########
@@ -25,13 +25,15 @@
from ..utils import get_const_tuple
from ..nn import dense_alter_layout
from .utils import target_has_vnni
+from .utils import target_has_amx
from .. import nn
-def check_vnni_applicable(x, y, allow_padding=False):
+def check_inst_applicable(x, y, allow_padding=False):
mcpu = tvm.target.Target.current().mcpu
+ cpu_avai = target_has_vnni(mcpu) or target_has_amx(mcpu)
return (
- target_has_vnni(mcpu)
+ cpu_avai
Review Comment:
* ~cpu_avai~ may sound strange, ```cpu_simd``` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] cbalint13 commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
cbalint13 commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1058190629
##########
python/tvm/topi/x86/tensor_intrin.py:
##########
@@ -348,3 +348,227 @@ def _instr(index):
binds={data: a_buffer, kernel: b_buffer},
default_buffer_params=buffer_params,
)
+
+
+def dot_32x128x32_u8s8s32_sapphirerapids(LDA):
+ """
+ Int8 dot product by every 16x64 elements using AMX-TMUL Sapphire Rapids instructions.
+ The tdpxxd instruction takes two tile of uint8 and int8 datatype -- data[16][64] and
+ kernel[1][16][16][4] -- and computes a dot product of data[16][16] in int32 datatype.
+
+ (Physically, to efficiently leveraging the tile register, we constructing a 2x2 tiles
+ matmul which performs 32x128x32 in total)
+
+ The pseudo code is as follows:
+ for(k=0; k<2; k++){
+ for(n=0; n<2; n++){
+ tileload64(tmm_b, B)
+ for(m=0; m<2; m++){
+ if(n==0)
+ tileload64(tmm_a, A)
Review Comment:
@masahi ,
I have a question related to your proposed here:
* Is there a TIR ```*_desc()``` equivalent for TOPI's body, reset, update ?
topi intrin decl: https://github.com/apache/tvm/blob/main/python/tvm/topi/x86/tensor_intrin.py#L140
* How the ```accum``` steps (very inner) can be succeeded by ```update``` (outer) or preceded by ```reset``` ?
I am looking for any TVM reference (code, example) for this, trying even finer granularity control of (auto)-tensorization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] Qianshui-Jiang commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
Qianshui-Jiang commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1058782556
##########
python/tvm/topi/x86/dense.py:
##########
@@ -373,6 +375,153 @@ def _callback(op):
return s
+def dense_amx_int8_compute(cfg, data, packed_w, bias=None):
+ """Compute for uint8 x int8 -> int32 dense"""
+ m, k = data.shape
+ n_o, _, n_i, _ = packed_w.shape
+ ak = te.reduce_axis((0, k), name="k")
+
+ C = te.compute(
+ (m, n_o * n_i),
+ lambda i, j: te.sum(
+ data[i, ak].astype("int32")
+ * packed_w[tvm.tir.indexdiv(j, 16), tvm.tir.indexdiv(ak, 4), j % 16, ak % 4].astype(
+ "int32"
+ ),
+ axis=ak,
+ ),
+ tag="dense_amx_int8",
+ attrs={"schedule_rule": "meta_schedule.dense_amx_int8"},
+ )
+
+ if bias is not None:
+ C = te.compute(C.shape, lambda i, j: C[i, j] + bias[j], tag=tag.BROADCAST)
+
+ return C
+
+
+def dense_amx_int8_schedule(cfg, s, C, O, do_parallel=True):
+ """Schedule dense compute using AMX TMUL instruction"""
+ # C: The output of GEMM
+ # O: The output of the fused op
+ def split_x(out):
+ default_x_split_factor1 = 32
+ default_x_split_factor2 = 2
+ default_x_split_factor3 = 2
+ default_x_split_factor4 = 2
+ a_x = s[out].op.axis[-2]
+
+ if cfg.is_fallback:
+ a_xo, a_xi = s[out].split(a_x, factor=default_x_split_factor1)
+ a_xo2, a_xo1 = s[out].split(a_xo, factor=default_x_split_factor2)
+ a_xo3, a_xo2 = s[out].split(a_xo2, factor=default_x_split_factor3)
+ a_xo4, a_xo3 = s[out].split(a_xo3, factor=default_x_split_factor4)
+ return [a_xo4, a_xo3, a_xo2, a_xo1, a_xi]
+
+ cfg.define_split("tile_x", a_x, num_outputs=5, filter=lambda x: x.size[-1] == 32)
+ return cfg["tile_x"].apply(s, out, a_x)
+
+ def split_y(out):
+ default_y_split_factor1 = 32
+ default_y_split_factor2 = 4
+ default_y_split_factor3 = 4
+ default_y_split_factor4 = 4
+ a_y = s[out].op.axis[-1]
+
+ if cfg.is_fallback:
+ a_yo1, a_yo = s[out].split(a_y, factor=default_y_split_factor1)
+ a_yo2, a_yo1 = s[out].split(a_yo1, factor=default_y_split_factor2)
+ a_yo3, a_yo2 = s[out].split(a_yo2, factor=default_y_split_factor3)
+ a_yo4, a_yo3 = s[out].split(a_yo3, factor=default_y_split_factor4)
+ return [a_yo4, a_yo3, a_yo2, a_yo1, a_yo]
+
+ cfg.define_split("tile_y", a_y, num_outputs=5, filter=lambda y: y.size[-1] == 32)
+ return cfg["tile_y"].apply(s, out, a_y)
+
+ def split_k(out, rd_axis):
+ default_k_split_factor1 = 128
+ default_k_split_factor2 = 2
+ default_k_split_factor3 = 2
+ default_k_split_factor4 = 2
+
+ if cfg.is_fallback:
+ a_ko, a_ki = s[out].split(rd_axis, factor=default_k_split_factor1)
+ a_ko2, a_ko1 = s[out].split(a_ko, factor=default_k_split_factor2)
+ a_ko3, a_ko2 = s[out].split(a_ko2, factor=default_k_split_factor3)
+ a_ko4, a_ko3 = s[out].split(a_ko3, factor=default_k_split_factor4)
+ return [a_ko4, a_ko3, a_ko2, a_ko1, a_ki]
+
+ cfg.define_split("tile_k", rd_axis, num_outputs=5, filter=lambda y: y.size[-1] == 128)
+ return cfg["tile_k"].apply(s, out, rd_axis)
+
+ a_x, a_y = C.op.axis
+ (a_k,) = C.op.reduce_axis
+ CF = s.cache_write(C, "amx.tmm")
+
+ a_x3, a_x2, a_x1, a_xo, a_xi = split_x(C)
+ a_y3, a_y2, a_y1, a_yo, a_yi = split_y(C)
+ s[C].reorder(a_x3, a_y3, a_x2, a_y2, a_x1, a_y1, a_xo, a_yo, a_xi, a_yi)
+
+ s[CF].compute_at(s[C], a_yo)
+
+ (a_k_f,) = CF.op.reduce_axis
+ a_x_f, a_y_f = CF.op.axis
+
+ a_xo_f, a_xi_f = s[CF].split(a_x_f, factor=32)
+
+ a_yo_f, a_yi_f = s[CF].split(a_y_f, factor=32)
+ a_k3_f, a_k2_f, a_k1_f, a_ko_f, a_ki_f = split_k(CF, a_k_f)
+ s[CF].reorder(a_k3_f, a_k2_f, a_k1_f, a_ko_f, a_xo_f, a_yo_f, a_ki_f, a_xi_f, a_yi_f)
+
+ (m, k) = CF.op.input_tensors[0].shape
+ (n, c, n_i, c_i) = CF.op.input_tensors[1].shape
+ n = n * n_i
+
+ s[CF].tensorize(a_ki_f, dot_32x128x32_u8s8s32_sapphirerapids(LDA=int(k)))
+ s[C].tensorize(a_xi, acc_32x32_int32_sapphirerapids(LDC=int(n)))
+
+ if C == O:
+ fused = s[O].fuse(a_x3, a_y3)
+ else:
+ a_y3, a_y2, a_y1, a_yr, a_yi = split_y(O)
+ a_x3, a_x2, a_x1, a_xr, a_xi = split_x(O)
+
+ s[O].reorder(a_y3, a_x3, a_y2, a_x2, a_y1, a_x1, a_yr, a_xr, a_yi, a_xi)
+ s[O].vectorize(a_xi)
+
+ fused = s[O].fuse(a_x3, a_y3)
+
+ if do_parallel:
+ s[O].parallel(fused)
+
+ return s, fused
+
+
+@autotvm.register_topi_compute("dense_amx_int8.x86")
+def dense_amx_int8(cfg, data, weight, bias=None, out_dtype=None):
+ """Compute for uint8 x int8 -> int32 dense"""
+ if out_dtype is None:
+ out_dtype = data.dtype
+ assert len(weight.shape) == 4
+ assert data.dtype == "uint8" and weight.dtype == "int8"
+ _, _, _, n_inner = get_const_tuple(weight.shape) # out_dim
+ assert n_inner == 4
+ return dense_amx_int8_compute(cfg, data, weight, bias)
+
+
+@autotvm.register_topi_schedule("dense_amx_int8.x86")
+def schedule_dense_amx_int8(cfg, outs):
+ """Create a schedule for dense_amx_int8"""
+ s = te.create_schedule([x.op for x in outs])
+
+ def _callback(op):
+ if "dense_amx_int8" in op.tag:
+ dense_amx_int8_schedule(cfg, s, op.output(0), outs[0])
+
+ traverse_inline(s, outs[0].op, _callback)
+ return s
Review Comment:
yep, already unified the AMX and VNNI interface in x86 strategy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
masahi commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1059735746
##########
python/tvm/topi/x86/dense.py:
##########
@@ -293,16 +324,13 @@ def dense_vnni_compute(cfg, X, packed_w, bias=None):
),
axis=ak,
),
- tag="dense_vnni",
- attrs={"schedule_rule": "dense_vnni"},
Review Comment:
Of course the test still works, because it was already written for VNNI. The point is that the name `dense_vnni` tells that the particular TE expression is meant for VNNI tensorization, in particular the weight is pre-packed appropriately. But a generic name like `dense_int8` doesn't provide such information.
So please revert that commit and restore and pass `attrs={"schedule_rule": "meta_schedule.x86.dense_vnni"}` when you create a compute expression for VNNI. `schedule_rule` is not relevant for AMX for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
masahi commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1059902201
##########
python/tvm/topi/x86/dense.py:
##########
@@ -293,16 +324,13 @@ def dense_vnni_compute(cfg, X, packed_w, bias=None):
),
axis=ak,
),
- tag="dense_vnni",
- attrs={"schedule_rule": "dense_vnni"},
Review Comment:
I suggest checking the target string in the op strategy, and create separate compute for VNNI or AMX (rather than using the same function, `dense_int8`, for both).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] Qianshui-Jiang commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
Qianshui-Jiang commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1058783171
##########
python/tvm/topi/x86/dense.py:
##########
@@ -373,6 +375,153 @@ def _callback(op):
return s
+def dense_amx_int8_compute(cfg, data, packed_w, bias=None):
Review Comment:
already unified AMX and VNNI to `dense_int8`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
masahi commented on PR #13642:
URL: https://github.com/apache/tvm/pull/13642#issuecomment-1365663717
@Qianshui-Jiang Can we test TVM-generated AMX code using the emulator (SDE)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] cbalint13 commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
cbalint13 commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1058190629
##########
python/tvm/topi/x86/tensor_intrin.py:
##########
@@ -348,3 +348,227 @@ def _instr(index):
binds={data: a_buffer, kernel: b_buffer},
default_buffer_params=buffer_params,
)
+
+
+def dot_32x128x32_u8s8s32_sapphirerapids(LDA):
+ """
+ Int8 dot product by every 16x64 elements using AMX-TMUL Sapphire Rapids instructions.
+ The tdpxxd instruction takes two tile of uint8 and int8 datatype -- data[16][64] and
+ kernel[1][16][16][4] -- and computes a dot product of data[16][16] in int32 datatype.
+
+ (Physically, to efficiently leveraging the tile register, we constructing a 2x2 tiles
+ matmul which performs 32x128x32 in total)
+
+ The pseudo code is as follows:
+ for(k=0; k<2; k++){
+ for(n=0; n<2; n++){
+ tileload64(tmm_b, B)
+ for(m=0; m<2; m++){
+ if(n==0)
+ tileload64(tmm_a, A)
Review Comment:
@masahi ,
I have a question related to your proposed here:
* Is there a TIR ```*_desc()``` equivalent for TOPI's body, reset, update ?
topi intrin decl: https://github.com/apache/tvm/blob/main/python/tvm/topi/x86/tensor_intrin.py#L140
* How (TIR) ```accum``` steps (very inner) can be succeeded by ```update``` (outer) or preceded by ```reset``` ?
I am looking for any TVM reference (code, example) for this, trying even finer granularity control of autotensorization with TIR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] cbalint13 commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
cbalint13 commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1058846644
##########
python/tvm/topi/x86/tensor_intrin.py:
##########
@@ -348,3 +348,227 @@ def _instr(index):
binds={data: a_buffer, kernel: b_buffer},
default_buffer_params=buffer_params,
)
+
+
+def dot_32x128x32_u8s8s32_sapphirerapids(LDA):
+ """
+ Int8 dot product by every 16x64 elements using AMX-TMUL Sapphire Rapids instructions.
+ The tdpxxd instruction takes two tile of uint8 and int8 datatype -- data[16][64] and
+ kernel[1][16][16][4] -- and computes a dot product of data[16][16] in int32 datatype.
+
+ (Physically, to efficiently leveraging the tile register, we constructing a 2x2 tiles
+ matmul which performs 32x128x32 in total)
+
+ The pseudo code is as follows:
+ for(k=0; k<2; k++){
+ for(n=0; n<2; n++){
+ tileload64(tmm_b, B)
+ for(m=0; m<2; m++){
+ if(n==0)
+ tileload64(tmm_a, A)
Review Comment:
@masahi ,
@Qianshui-Jiang ,
Don't want hold up the PR, I think it is fine to merge with this start AMX implementation.
I leave here a possible way to workaround the rigidity of LLVM's AMX related intrinsic declaration:
* Use ```tvm.tir.call_extern``` insead of ```tvm.tir.call_llvm_intrin``` like [here](https://github.com/apache/tvm/blob/8551a5c71faaa2eedc8b1c49a39be0fe4688c021/python/tvm/topi/arm_cpu/mprofile/dsp/micro_kernel/gemm.py#L107-L109) .
* You can than pass argument (e.g. register number, etc ...) into the custom called function.
* You can use the argument inside that function and emit the AMX ISA as bytes: [ __asm(.word)](https://github.com/cbalint13/OLIMP/blob/main/demo/icebreaker/src/rv32-custom.h#L52-L53)
This is how it is done to emit custom part of RISC-V ISA, that is also too rigid using just the LLVM declarations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
masahi commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1059588708
##########
python/tvm/topi/x86/dense.py:
##########
@@ -293,16 +324,13 @@ def dense_vnni_compute(cfg, X, packed_w, bias=None):
),
axis=ak,
),
- tag="dense_vnni",
- attrs={"schedule_rule": "dense_vnni"},
Review Comment:
This shouldn't be removed, it is used here https://github.com/apache/tvm/blob/b16a64d6edb9fd1a014fc51995dff7d0e2f4c84e/tests/python/unittest/test_meta_schedule_vnni_integration.py#L198.
Since this only affects MetaSchedule, you don't have to provide this value for AMX. So only when `dense_int8_compute` is called for VNNI, you need to provide this attribute.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] cbalint13 commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
cbalint13 commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1056879622
##########
tests/python/contrib/test_gemm_acc32_vnni.py:
##########
@@ -115,5 +115,5 @@ def verify(target="llvm -mcpu=cascadelake"):
# The test requires Cascade Lake and newer Intel machines to generate the
# correct AVX512 VNNI instruction. So, disabling the test.
- # test_fc_int8_acc32()
+ # test_fc_int8_acc32()``
Review Comment:
* spurious change ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] cbalint13 commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
cbalint13 commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1058846644
##########
python/tvm/topi/x86/tensor_intrin.py:
##########
@@ -348,3 +348,227 @@ def _instr(index):
binds={data: a_buffer, kernel: b_buffer},
default_buffer_params=buffer_params,
)
+
+
+def dot_32x128x32_u8s8s32_sapphirerapids(LDA):
+ """
+ Int8 dot product by every 16x64 elements using AMX-TMUL Sapphire Rapids instructions.
+ The tdpxxd instruction takes two tile of uint8 and int8 datatype -- data[16][64] and
+ kernel[1][16][16][4] -- and computes a dot product of data[16][16] in int32 datatype.
+
+ (Physically, to efficiently leveraging the tile register, we constructing a 2x2 tiles
+ matmul which performs 32x128x32 in total)
+
+ The pseudo code is as follows:
+ for(k=0; k<2; k++){
+ for(n=0; n<2; n++){
+ tileload64(tmm_b, B)
+ for(m=0; m<2; m++){
+ if(n==0)
+ tileload64(tmm_a, A)
Review Comment:
@masahi ,
@Qianshui-Jiang ,
Don't want hold up the PR, I think it is fine to merge with this start AMX implementation.
I leave here a possible way to workaround the rigidity of LLVM's AMX related intrinsic declaration:
* Use ```tvm.tir.call_extern``` insead of ```tvm.tir.call_llvm_intrin``` like [here](https://github.com/apache/tvm/blob/8551a5c71faaa2eedc8b1c49a39be0fe4688c021/python/tvm/topi/arm_cpu/mprofile/dsp/micro_kernel/gemm.py#L107-L109) .
* You can than pass argument (e.g. register number, etc ...) into the custom called function.
* You can use the argument inside that function and emit the AMX ISA as bytes: [ __asm(.word packed())](https://github.com/cbalint13/OLIMP/blob/main/demo/icebreaker/src/rv32-custom.h#L52-L53)
This is how it is done to emit custom part of RISC-V ISA, that is also too rigid using just the LLVM declarations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] Qianshui-Jiang commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
Qianshui-Jiang commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1059227039
##########
python/tvm/topi/x86/tensor_intrin.py:
##########
@@ -348,3 +348,227 @@ def _instr(index):
binds={data: a_buffer, kernel: b_buffer},
default_buffer_params=buffer_params,
)
+
+
+def dot_32x128x32_u8s8s32_sapphirerapids(LDA):
+ """
+ Int8 dot product by every 16x64 elements using AMX-TMUL Sapphire Rapids instructions.
+ The tdpxxd instruction takes two tile of uint8 and int8 datatype -- data[16][64] and
+ kernel[1][16][16][4] -- and computes a dot product of data[16][16] in int32 datatype.
+
+ (Physically, to efficiently leveraging the tile register, we constructing a 2x2 tiles
+ matmul which performs 32x128x32 in total)
+
+ The pseudo code is as follows:
+ for(k=0; k<2; k++){
+ for(n=0; n<2; n++){
+ tileload64(tmm_b, B)
+ for(m=0; m<2; m++){
+ if(n==0)
+ tileload64(tmm_a, A)
Review Comment:
@cbalint13, Thanks a lot for your advices.
Regarding that in TVM we already have DNNL codegen and some external call to DNNL, which is an off-the -shelf compute lib leverages SIMD ISA transparently.
The main focus in this PR is to add more native capability for TVM to generate AMX instructions.
AFAIK, the LLVM features for AMX are also moving forward such as register allocation for intrinsics and auto vectorization.
Hence we choose to call LLVM intrinsics directly and hope to have more flexibility in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] Qianshui-Jiang commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
Qianshui-Jiang commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1059227039
##########
python/tvm/topi/x86/tensor_intrin.py:
##########
@@ -348,3 +348,227 @@ def _instr(index):
binds={data: a_buffer, kernel: b_buffer},
default_buffer_params=buffer_params,
)
+
+
+def dot_32x128x32_u8s8s32_sapphirerapids(LDA):
+ """
+ Int8 dot product by every 16x64 elements using AMX-TMUL Sapphire Rapids instructions.
+ The tdpxxd instruction takes two tile of uint8 and int8 datatype -- data[16][64] and
+ kernel[1][16][16][4] -- and computes a dot product of data[16][16] in int32 datatype.
+
+ (Physically, to efficiently leveraging the tile register, we constructing a 2x2 tiles
+ matmul which performs 32x128x32 in total)
+
+ The pseudo code is as follows:
+ for(k=0; k<2; k++){
+ for(n=0; n<2; n++){
+ tileload64(tmm_b, B)
+ for(m=0; m<2; m++){
+ if(n==0)
+ tileload64(tmm_a, A)
Review Comment:
@cbalint13, Thanks a lot for your advices.
Regarding that in TVM we already have DNNL codegen and some external call to DNNL, which is an off-the -shelf compute lib leverages SIMD ISA transparently.
The main focus in this PR is to add more native capability for TVM to generate AMX instructions.
AFAIK, the LLVM features for AMX are also moving forward such as register allocation for intrinsics and auto vectorization. Hence we choose to call LLVM intrinsics directly and hope to have more flexibility in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] Qianshui-Jiang commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
Qianshui-Jiang commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1056955524
##########
python/tvm/topi/x86/dense_alter_op.py:
##########
@@ -25,13 +25,15 @@
from ..utils import get_const_tuple
from ..nn import dense_alter_layout
from .utils import target_has_vnni
+from .utils import target_has_amx
from .. import nn
-def check_vnni_applicable(x, y, allow_padding=False):
+def check_inst_applicable(x, y, allow_padding=False):
mcpu = tvm.target.Target.current().mcpu
+ cpu_avai = target_has_vnni(mcpu) or target_has_amx(mcpu)
return (
- target_has_vnni(mcpu)
+ cpu_avai
Review Comment:
yep, replaced `cpu_avai` to `simd_avai` to check if vnni or amx was availiable in this target.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] Qianshui-Jiang commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
Qianshui-Jiang commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1059650814
##########
python/tvm/topi/x86/dense.py:
##########
@@ -293,16 +324,13 @@ def dense_vnni_compute(cfg, X, packed_w, bias=None):
),
axis=ak,
),
- tag="dense_vnni",
- attrs={"schedule_rule": "dense_vnni"},
Review Comment:
@masahi Is this case only use the x86 int8 compute method and inject a particular TIR scheduling? Can we just change the attribute `dense_vnni` to `dense_int8` which used here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] Qianshui-Jiang commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
Qianshui-Jiang commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1061114811
##########
python/tvm/relay/op/strategy/x86.py:
##########
@@ -591,18 +591,17 @@ def dense_strategy_cpu(attrs, inputs, out_type, target):
def dense_pack_strategy_cpu(attrs, inputs, out_type, target):
"""dense_pack x86 strategy"""
strategy = _op.OpStrategy()
-
if (
inputs[0].dtype == "uint8"
and inputs[1].dtype == "int8"
and out_type.dtype == "int32"
and attrs["weight_layout"] == "NC16n4c"
):
Review Comment:
@masahi already added the target checking, seems generic compute wrapper didn't have the interface. but it's the same if checking inside of `dense_int8`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] Qianshui-Jiang commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
Qianshui-Jiang commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1062065916
##########
apps/microtvm/zephyr/template_project/qemu-hack/qemu-system-arm:
##########
@@ -1 +0,0 @@
-qemu-system-i386
Review Comment:
@masahi by misoperation, already removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
masahi commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1057990584
##########
python/tvm/topi/x86/dense.py:
##########
@@ -373,6 +375,153 @@ def _callback(op):
return s
+def dense_amx_int8_compute(cfg, data, packed_w, bias=None):
Review Comment:
This can be shared with the vnni compute.
##########
python/tvm/topi/x86/dense.py:
##########
@@ -373,6 +375,153 @@ def _callback(op):
return s
+def dense_amx_int8_compute(cfg, data, packed_w, bias=None):
+ """Compute for uint8 x int8 -> int32 dense"""
+ m, k = data.shape
+ n_o, _, n_i, _ = packed_w.shape
+ ak = te.reduce_axis((0, k), name="k")
+
+ C = te.compute(
+ (m, n_o * n_i),
+ lambda i, j: te.sum(
+ data[i, ak].astype("int32")
+ * packed_w[tvm.tir.indexdiv(j, 16), tvm.tir.indexdiv(ak, 4), j % 16, ak % 4].astype(
+ "int32"
+ ),
+ axis=ak,
+ ),
+ tag="dense_amx_int8",
+ attrs={"schedule_rule": "meta_schedule.dense_amx_int8"},
+ )
+
+ if bias is not None:
+ C = te.compute(C.shape, lambda i, j: C[i, j] + bias[j], tag=tag.BROADCAST)
+
+ return C
+
+
+def dense_amx_int8_schedule(cfg, s, C, O, do_parallel=True):
+ """Schedule dense compute using AMX TMUL instruction"""
+ # C: The output of GEMM
+ # O: The output of the fused op
+ def split_x(out):
+ default_x_split_factor1 = 32
+ default_x_split_factor2 = 2
+ default_x_split_factor3 = 2
+ default_x_split_factor4 = 2
+ a_x = s[out].op.axis[-2]
+
+ if cfg.is_fallback:
+ a_xo, a_xi = s[out].split(a_x, factor=default_x_split_factor1)
+ a_xo2, a_xo1 = s[out].split(a_xo, factor=default_x_split_factor2)
+ a_xo3, a_xo2 = s[out].split(a_xo2, factor=default_x_split_factor3)
+ a_xo4, a_xo3 = s[out].split(a_xo3, factor=default_x_split_factor4)
+ return [a_xo4, a_xo3, a_xo2, a_xo1, a_xi]
+
+ cfg.define_split("tile_x", a_x, num_outputs=5, filter=lambda x: x.size[-1] == 32)
+ return cfg["tile_x"].apply(s, out, a_x)
+
+ def split_y(out):
+ default_y_split_factor1 = 32
+ default_y_split_factor2 = 4
+ default_y_split_factor3 = 4
+ default_y_split_factor4 = 4
+ a_y = s[out].op.axis[-1]
+
+ if cfg.is_fallback:
+ a_yo1, a_yo = s[out].split(a_y, factor=default_y_split_factor1)
+ a_yo2, a_yo1 = s[out].split(a_yo1, factor=default_y_split_factor2)
+ a_yo3, a_yo2 = s[out].split(a_yo2, factor=default_y_split_factor3)
+ a_yo4, a_yo3 = s[out].split(a_yo3, factor=default_y_split_factor4)
+ return [a_yo4, a_yo3, a_yo2, a_yo1, a_yo]
+
+ cfg.define_split("tile_y", a_y, num_outputs=5, filter=lambda y: y.size[-1] == 32)
+ return cfg["tile_y"].apply(s, out, a_y)
+
+ def split_k(out, rd_axis):
+ default_k_split_factor1 = 128
+ default_k_split_factor2 = 2
+ default_k_split_factor3 = 2
+ default_k_split_factor4 = 2
+
+ if cfg.is_fallback:
+ a_ko, a_ki = s[out].split(rd_axis, factor=default_k_split_factor1)
+ a_ko2, a_ko1 = s[out].split(a_ko, factor=default_k_split_factor2)
+ a_ko3, a_ko2 = s[out].split(a_ko2, factor=default_k_split_factor3)
+ a_ko4, a_ko3 = s[out].split(a_ko3, factor=default_k_split_factor4)
+ return [a_ko4, a_ko3, a_ko2, a_ko1, a_ki]
+
+ cfg.define_split("tile_k", rd_axis, num_outputs=5, filter=lambda y: y.size[-1] == 128)
+ return cfg["tile_k"].apply(s, out, rd_axis)
+
+ a_x, a_y = C.op.axis
+ (a_k,) = C.op.reduce_axis
+ CF = s.cache_write(C, "amx.tmm")
+
+ a_x3, a_x2, a_x1, a_xo, a_xi = split_x(C)
+ a_y3, a_y2, a_y1, a_yo, a_yi = split_y(C)
+ s[C].reorder(a_x3, a_y3, a_x2, a_y2, a_x1, a_y1, a_xo, a_yo, a_xi, a_yi)
+
+ s[CF].compute_at(s[C], a_yo)
+
+ (a_k_f,) = CF.op.reduce_axis
+ a_x_f, a_y_f = CF.op.axis
+
+ a_xo_f, a_xi_f = s[CF].split(a_x_f, factor=32)
+
+ a_yo_f, a_yi_f = s[CF].split(a_y_f, factor=32)
+ a_k3_f, a_k2_f, a_k1_f, a_ko_f, a_ki_f = split_k(CF, a_k_f)
+ s[CF].reorder(a_k3_f, a_k2_f, a_k1_f, a_ko_f, a_xo_f, a_yo_f, a_ki_f, a_xi_f, a_yi_f)
+
+ (m, k) = CF.op.input_tensors[0].shape
+ (n, c, n_i, c_i) = CF.op.input_tensors[1].shape
+ n = n * n_i
+
+ s[CF].tensorize(a_ki_f, dot_32x128x32_u8s8s32_sapphirerapids(LDA=int(k)))
+ s[C].tensorize(a_xi, acc_32x32_int32_sapphirerapids(LDC=int(n)))
+
+ if C == O:
+ fused = s[O].fuse(a_x3, a_y3)
+ else:
+ a_y3, a_y2, a_y1, a_yr, a_yi = split_y(O)
+ a_x3, a_x2, a_x1, a_xr, a_xi = split_x(O)
+
+ s[O].reorder(a_y3, a_x3, a_y2, a_x2, a_y1, a_x1, a_yr, a_xr, a_yi, a_xi)
+ s[O].vectorize(a_xi)
+
+ fused = s[O].fuse(a_x3, a_y3)
+
+ if do_parallel:
+ s[O].parallel(fused)
+
+ return s, fused
+
+
+@autotvm.register_topi_compute("dense_amx_int8.x86")
+def dense_amx_int8(cfg, data, weight, bias=None, out_dtype=None):
+ """Compute for uint8 x int8 -> int32 dense"""
+ if out_dtype is None:
+ out_dtype = data.dtype
+ assert len(weight.shape) == 4
+ assert data.dtype == "uint8" and weight.dtype == "int8"
+ _, _, _, n_inner = get_const_tuple(weight.shape) # out_dim
+ assert n_inner == 4
+ return dense_amx_int8_compute(cfg, data, weight, bias)
+
+
+@autotvm.register_topi_schedule("dense_amx_int8.x86")
+def schedule_dense_amx_int8(cfg, outs):
+ """Create a schedule for dense_amx_int8"""
+ s = te.create_schedule([x.op for x in outs])
+
+ def _callback(op):
+ if "dense_amx_int8" in op.tag:
+ dense_amx_int8_schedule(cfg, s, op.output(0), outs[0])
+
+ traverse_inline(s, outs[0].op, _callback)
+ return s
Review Comment:
For the two functions above, try removing dup with VNNI ones.
##########
python/tvm/topi/x86/tensor_intrin.py:
##########
@@ -348,3 +348,227 @@ def _instr(index):
binds={data: a_buffer, kernel: b_buffer},
default_buffer_params=buffer_params,
)
+
+
+def dot_32x128x32_u8s8s32_sapphirerapids(LDA):
+ """
+ Int8 dot product by every 16x64 elements using AMX-TMUL Sapphire Rapids instructions.
+ The tdpxxd instruction takes two tile of uint8 and int8 datatype -- data[16][64] and
+ kernel[1][16][16][4] -- and computes a dot product of data[16][16] in int32 datatype.
+
+ (Physically, to efficiently leveraging the tile register, we constructing a 2x2 tiles
+ matmul which performs 32x128x32 in total)
+
+ The pseudo code is as follows:
+ for(k=0; k<2; k++){
+ for(n=0; n<2; n++){
+ tileload64(tmm_b, B)
+ for(m=0; m<2; m++){
+ if(n==0)
+ tileload64(tmm_a, A)
Review Comment:
Have you considered tensorizing at finer granularity? Meaning, instead of tensoring the whole load + compute as one tensor intrin, tensorize each load and compute separately. That could make the hard coded outer-loop factors (2 in this intrin) tunable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] cbalint13 commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
cbalint13 commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1058190629
##########
python/tvm/topi/x86/tensor_intrin.py:
##########
@@ -348,3 +348,227 @@ def _instr(index):
binds={data: a_buffer, kernel: b_buffer},
default_buffer_params=buffer_params,
)
+
+
+def dot_32x128x32_u8s8s32_sapphirerapids(LDA):
+ """
+ Int8 dot product by every 16x64 elements using AMX-TMUL Sapphire Rapids instructions.
+ The tdpxxd instruction takes two tile of uint8 and int8 datatype -- data[16][64] and
+ kernel[1][16][16][4] -- and computes a dot product of data[16][16] in int32 datatype.
+
+ (Physically, to efficiently leveraging the tile register, we constructing a 2x2 tiles
+ matmul which performs 32x128x32 in total)
+
+ The pseudo code is as follows:
+ for(k=0; k<2; k++){
+ for(n=0; n<2; n++){
+ tileload64(tmm_b, B)
+ for(m=0; m<2; m++){
+ if(n==0)
+ tileload64(tmm_a, A)
Review Comment:
@masahi ,
I have a question related to your proposed here:
* Is there a TIR _desc equivalent for TOPI's body, reset, update ?
topi intrin decl: https://github.com/apache/tvm/blob/main/python/tvm/topi/x86/tensor_intrin.py#L140
* How the ```accum``` steps (very inner) can be succeeded by ```update``` (outer) or preceded by ```reset``` ?
I am looking for any TVM reference (code, example) for this, trying even finer granularity control of tensorization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] cbalint13 commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
cbalint13 commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1058287354
##########
python/tvm/topi/x86/tensor_intrin.py:
##########
@@ -348,3 +348,227 @@ def _instr(index):
binds={data: a_buffer, kernel: b_buffer},
default_buffer_params=buffer_params,
)
+
+
+def dot_32x128x32_u8s8s32_sapphirerapids(LDA):
+ """
+ Int8 dot product by every 16x64 elements using AMX-TMUL Sapphire Rapids instructions.
+ The tdpxxd instruction takes two tile of uint8 and int8 datatype -- data[16][64] and
+ kernel[1][16][16][4] -- and computes a dot product of data[16][16] in int32 datatype.
+
+ (Physically, to efficiently leveraging the tile register, we constructing a 2x2 tiles
+ matmul which performs 32x128x32 in total)
+
+ The pseudo code is as follows:
+ for(k=0; k<2; k++){
+ for(n=0; n<2; n++){
+ tileload64(tmm_b, B)
+ for(m=0; m<2; m++){
+ if(n==0)
+ tileload64(tmm_a, A)
Review Comment:
> If you are asking about tensorization in TIR, there is no need to have separate init, update etc. See https://github.com/apache/tvm/blob/main/python/tvm/tir/tensor_intrin/x86.py. The accum buffer is implicitly zero-initialized and we can direct do update via `+=`.
@masahi ,
Thanks a lot ! I've noticed now "+=" for ```update``` purpose in TIR description.
Will also look @ MultiLevelTiling() (cuda only) idea for custom ISA having special ```init```, ```load_*```, ```store```.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
masahi commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1058267062
##########
python/tvm/topi/x86/tensor_intrin.py:
##########
@@ -348,3 +348,227 @@ def _instr(index):
binds={data: a_buffer, kernel: b_buffer},
default_buffer_params=buffer_params,
)
+
+
+def dot_32x128x32_u8s8s32_sapphirerapids(LDA):
+ """
+ Int8 dot product by every 16x64 elements using AMX-TMUL Sapphire Rapids instructions.
+ The tdpxxd instruction takes two tile of uint8 and int8 datatype -- data[16][64] and
+ kernel[1][16][16][4] -- and computes a dot product of data[16][16] in int32 datatype.
+
+ (Physically, to efficiently leveraging the tile register, we constructing a 2x2 tiles
+ matmul which performs 32x128x32 in total)
+
+ The pseudo code is as follows:
+ for(k=0; k<2; k++){
+ for(n=0; n<2; n++){
+ tileload64(tmm_b, B)
+ for(m=0; m<2; m++){
+ if(n==0)
+ tileload64(tmm_a, A)
Review Comment:
If you are asking about tensorization in TIR, there is no need to have separate init, update etc. See https://github.com/apache/tvm/blob/main/python/tvm/tir/tensor_intrin/x86.py. The accum buffer is implicitly zero-initialized and we can direct do update via `+=`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
masahi commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1059902491
##########
python/tvm/relay/op/strategy/x86.py:
##########
@@ -591,18 +591,17 @@ def dense_strategy_cpu(attrs, inputs, out_type, target):
def dense_pack_strategy_cpu(attrs, inputs, out_type, target):
"""dense_pack x86 strategy"""
strategy = _op.OpStrategy()
-
if (
inputs[0].dtype == "uint8"
and inputs[1].dtype == "int8"
and out_type.dtype == "int32"
and attrs["weight_layout"] == "NC16n4c"
):
Review Comment:
Let's check the target mcpu here and pass VNNI or AMX attribute to `dense_int8`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] Qianshui-Jiang commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
Qianshui-Jiang commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1056955080
##########
tests/python/contrib/test_gemm_acc32_vnni.py:
##########
@@ -115,5 +115,5 @@ def verify(target="llvm -mcpu=cascadelake"):
# The test requires Cascade Lake and newer Intel machines to generate the
# correct AVX512 VNNI instruction. So, disabling the test.
- # test_fc_int8_acc32()
+ # test_fc_int8_acc32()``
Review Comment:
thank you for comment, already fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] Qianshui-Jiang commented on pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
Qianshui-Jiang commented on PR #13642:
URL: https://github.com/apache/tvm/pull/13642#issuecomment-1365954741
@masahi, BTW, SDE would be very slow, the repeat times of time_evaluator should be small.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] Qianshui-Jiang commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
Qianshui-Jiang commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1058782058
##########
python/tvm/topi/x86/tensor_intrin.py:
##########
@@ -348,3 +348,227 @@ def _instr(index):
binds={data: a_buffer, kernel: b_buffer},
default_buffer_params=buffer_params,
)
+
+
+def dot_32x128x32_u8s8s32_sapphirerapids(LDA):
+ """
+ Int8 dot product by every 16x64 elements using AMX-TMUL Sapphire Rapids instructions.
+ The tdpxxd instruction takes two tile of uint8 and int8 datatype -- data[16][64] and
+ kernel[1][16][16][4] -- and computes a dot product of data[16][16] in int32 datatype.
+
+ (Physically, to efficiently leveraging the tile register, we constructing a 2x2 tiles
+ matmul which performs 32x128x32 in total)
+
+ The pseudo code is as follows:
+ for(k=0; k<2; k++){
+ for(n=0; n<2; n++){
+ tileload64(tmm_b, B)
+ for(m=0; m<2; m++){
+ if(n==0)
+ tileload64(tmm_a, A)
Review Comment:
> Have you considered tensorizing at finer granularity? Meaning, instead of tensoring the whole load + compute as one tensor intrin, tensorize each load and compute separately. That could make the hard coded outer-loop factors (2 in this intrin) tunable.
@masahi Yes we considered to make it works like tensor core which mentioned here: https://github.com/apache/tvm/issues/4052, but the problem is AMX intrinsics in LLVM need user to assigned register(tmm) number by hand craft. If we seperate the load and compute, we have to leave the register assginment to schedule method, that would bring much redundancy work. So finnally we decide to follow BRGEMM and leave the whole smallest comute unit for tensorizing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Posted by GitBox <gi...@apache.org>.
masahi commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1059717256
##########
python/tvm/topi/x86/dense.py:
##########
@@ -293,16 +324,13 @@ def dense_vnni_compute(cfg, X, packed_w, bias=None):
),
axis=ak,
),
- tag="dense_vnni",
- attrs={"schedule_rule": "dense_vnni"},
Review Comment:
Yes, but it is important that we specify that we use this compute for VNNI. If the schedule rule annotation only says "dense_int8", we don't know which intrinsic to tensorize this compute with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org