You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by "lhutton1 (via GitHub)" <gi...@apache.org> on 2024/05/03 09:27:43 UTC

[PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

lhutton1 opened a new pull request, #16966:
URL: https://github.com/apache/tvm/pull/16966

   ### Representation
   This commit extends `BufferLoad` and `BufferStore` to accept a predicate mask argument indicating which lanes in a vectorized buffer load/store should be read/written.
   
   As a simple example, we can load all lanes:
   ```
   tir.BufferLoad(buf, [tir.Ramp(0, 1, 8)], predicate=tir.Broadcast(1, 8))
   ```
   
   Or disable loading all lanes:
   ```
   tir.BufferLoad(buf, [tir.Ramp(0, 1, 8)], predicate=tir.Broadcast(0, 8))
   ```
   
   In TVMScript, buffer loads and stores are currently displayed using a "short-hand" notation e.g. `A[0:4]`, but there was no clear path for extending this notation to support predicates. Therefore, a "long-hand" notation is introduced e.g. `A.load([T.Ramp(0, 1, 4)], predicate=...)`. The TVMScript printer falls back to the long-hand notation whenever predicates are specified.
   
   ### Creation
   Buffer-level predication becomes more motivating when combined with the `tir.get_active_lane_mask` intrinsic. It can be used to mask off lanes when the vectorized axis is not divisible by the vector length. A detailed example and rationale can be found in the [RFC](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0104-scalable-vectors-in-tir.md#predication).
   
   Predicated buffer load/stores are created in the `VectorizeLoop` pass via `TryPredicateBufferAccesses`. This pass aims to convert block-level predicates e.g.
   ```
   for i_0 in T.serial(4):
       for i_1 in T.vectorized(4):
           if i_0 * 4 + i_1 < 14:
               B[i_0 * 4 + i_1] = A[i_0 * 4 + i_1] + 1.0
   ```
   to buffer-level predicates, e.g.
   ```
   for i_0 in T.serial(4):
       predicate = T.get_active_lane_mask("int1x4", i_0 * 4, 14)
       A_load = T.meta_var(A.load([T.Ramp(i_0 * 4, 1, 4)], predicate=predicate))
       B.store(A_load, [T.Ramp(i_0 * 4, 1, 4)], predicate=predicate)
   ```
   It takes a conservative approach for now, focussing only on expressions produced by the split scheduling primitive, but more complex expressions could be supported in the future.
   
   `TryPredicateBufferAccesses` can be explicitly enabled/disabled with the `tir.enable_buffer_level_predication` pass context option. By default it will be disabled, unless the target supports SVE, in which case it will be enabled by default.
   
   Note: this commit depends on https://github.com/apache/tvm/pull/16965, so also contains the contents of https://github.com/apache/tvm/pull/16965.
   
   Co-authored-by: Elen Kalda <el...@arm.com>
   Co-authored-by: Neil Hickey <ne...@arm.com>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

Posted by "lhutton1 (via GitHub)" <gi...@apache.org>.
lhutton1 commented on code in PR #16966:
URL: https://github.com/apache/tvm/pull/16966#discussion_r1606010962


##########
include/tvm/tir/buffer.h:
##########
@@ -209,14 +209,20 @@ class Buffer : public ObjectRef {
    * \brief Create an Expr that does a vector load at begin index.
    * \param begin The beginning index
    * \param dtype The data type to be loaded.
+   * \param predicate A vector mask of boolean values indicating which lanes of a vector are to be
+   * stored. The number lanes of the mask must be equal to the number of lanes in value.

Review Comment:
   Oops thanks, result of copy and pasting



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

Posted by "lhutton1 (via GitHub)" <gi...@apache.org>.
lhutton1 commented on PR #16966:
URL: https://github.com/apache/tvm/pull/16966#issuecomment-2098219142

   cc @ekalda @tqchen @Lunderberg @cbalint13 @Anndrey24 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

Posted by "lhutton1 (via GitHub)" <gi...@apache.org>.
lhutton1 commented on PR #16966:
URL: https://github.com/apache/tvm/pull/16966#issuecomment-2124698491

   friendly ping @Lunderberg if you have some free time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

Posted by "lhutton1 (via GitHub)" <gi...@apache.org>.
lhutton1 commented on code in PR #16966:
URL: https://github.com/apache/tvm/pull/16966#discussion_r1600019929


##########
src/tir/ir/stmt.cc:
##########
@@ -458,7 +458,8 @@ TVM_REGISTER_GLOBAL("tir.Evaluate").set_body_typed([](PrimExpr value, Span span)
 TVM_REGISTER_NODE_TYPE(EvaluateNode);
 
 // BufferStore
-BufferStore::BufferStore(Buffer buffer, PrimExpr value, Array<PrimExpr> indices, Span span) {
+BufferStore::BufferStore(Buffer buffer, PrimExpr value, Array<PrimExpr> indices, PrimExpr predicate,
+                         Span span) {
   ICHECK_EQ(buffer->shape.size(), indices.size())

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

Posted by "Lunderberg (via GitHub)" <gi...@apache.org>.
Lunderberg commented on PR #16966:
URL: https://github.com/apache/tvm/pull/16966#issuecomment-2108122470

   No problem, and thank you on the revisions!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

Posted by "lhutton1 (via GitHub)" <gi...@apache.org>.
lhutton1 commented on code in PR #16966:
URL: https://github.com/apache/tvm/pull/16966#discussion_r1600003875


##########
tests/python/codegen/test_target_codegen_aarch64.py:
##########
@@ -700,5 +700,31 @@ def before(a: T.handle):
     assert "get.active.lane.mask" in ll
 
 
+@pytest.mark.skipif(

Review Comment:
   Done - I wasn't able to check all targets locally, so I'm hoping they pass CI here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

Posted by "lhutton1 (via GitHub)" <gi...@apache.org>.
lhutton1 commented on code in PR #16966:
URL: https://github.com/apache/tvm/pull/16966#discussion_r1598578740


##########
src/tir/transforms/vectorize_loop.cc:
##########
@@ -72,6 +72,126 @@ inline PrimExpr BroadcastTo(PrimExpr e, int lanes, bool is_scalable) {
   return Broadcast(e, CreateNewLanes(is_scalable, lanes));
 }
 
+bool EnableBufferLevelPredication() {

Review Comment:
   Thanks, that makes sense. Do you know if there is any general infrastructure for keep track of the current target (which takes into account this override functionality) from within a pass? Otherwise I feel we will be duplicating this functionality in multiple places. I was thinking something similar to: `LexicalOnDeviceMixin` (assuming I understood it correctly)



##########
src/target/llvm/codegen_llvm.cc:
##########
@@ -1768,11 +1774,17 @@ llvm::Value* CodeGenLLVM::VisitExpr_(const BufferLoadNode* op) {
 
   std::vector<llvm::Value*> loads;
 
-  auto make_load = [this, &loads](TypedPointer buffer_ptr, int /* subelement_i */, int alignment,
-                                  bool is_volatile) {
+  auto make_load = [this, &loads](TypedPointer buffer_ptr, int /* subelement_i */,
+                                  llvm::Value* predicate, int alignment, bool is_volatile) {
 #if TVM_LLVM_VERSION >= 110
-    auto load = builder_->CreateAlignedLoad(buffer_ptr.type, buffer_ptr.addr,
-                                            llvm::Align(alignment), is_volatile);
+    llvm::Instruction* load = nullptr;
+    if (predicate != NULL) {
+      load = builder_->CreateMaskedLoad(buffer_ptr.type, buffer_ptr.addr, llvm::Align(alignment),
+                                        predicate);

Review Comment:
   Good point, thanks. I took a look at MaskedLoad in LLVM and it doesn't seem possible to create a volatile masked load, I'll throw an exception for now.



##########
python/tvm/tir/buffer.py:
##########
@@ -141,6 +141,57 @@ def vstore(self, begin, value):
         begin = (begin,) if isinstance(begin, (int, PrimExpr)) else begin
         return _ffi_api.BufferVStore(self, begin, value)  # type: ignore
 
+    def load(self, indices, predicate=None):

Review Comment:
   We did look at reusing these but decided against it since the vload/vstore api doesn't let you specify the "indices" argument of the underlying buffer load/store which can limit expressing more complex (e.g. non-consecutive) accesses
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

Posted by "lhutton1 (via GitHub)" <gi...@apache.org>.
lhutton1 commented on PR #16966:
URL: https://github.com/apache/tvm/pull/16966#issuecomment-2122079827

   @tvm-bot rerun


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

Posted by "tqchen (via GitHub)" <gi...@apache.org>.
tqchen commented on code in PR #16966:
URL: https://github.com/apache/tvm/pull/16966#discussion_r1618046951


##########
tests/python/codegen/test_target_codegen_aarch64.py:
##########
@@ -780,5 +780,31 @@ def before(a: T.handle):
     assert "get.active.lane.mask" in ll
 
 
+@pytest.mark.skipif(
+    llvm_version_major() < 11,
+    reason="Vscale and get.active.lane.mask are not supported in earlier versions of LLVM",
+)
+def test_predicated_scalable_buffer():
+    target = "llvm -mtriple=aarch64-linux-gnu -mattr=+sve"
+
+    @T.prim_func
+    def before(a: T.handle, b: T.handle):
+        A = T.match_buffer(a, (16,), "float32")
+        B = T.match_buffer(b, (16,), "float32")
+        T.func_attr({"global_symbol": "main", "tir.noalias": True})
+        for i_0 in T.serial(T.ceildiv(16, 4 * T.vscale())):
+            for i_1 in T.vectorized(4 * T.vscale()):
+                if i_0 * 4 * T.vscale() + i_1 < 14:
+                    B[i_0 * 4 * T.vscale() + i_1] = A[i_0 * 4 * T.vscale() + i_1] + 1.0
+
+    with tvm.target.Target(target):

Review Comment:
   we should avoid this style of code(this was created for legacy usage), instead, attach the target attribute to the PrimFunc itself



##########
tests/python/codegen/test_target_codegen_aarch64.py:
##########
@@ -780,5 +780,31 @@ def before(a: T.handle):
     assert "get.active.lane.mask" in ll
 
 
+@pytest.mark.skipif(
+    llvm_version_major() < 11,
+    reason="Vscale and get.active.lane.mask are not supported in earlier versions of LLVM",
+)
+def test_predicated_scalable_buffer():
+    target = "llvm -mtriple=aarch64-linux-gnu -mattr=+sve"
+
+    @T.prim_func
+    def before(a: T.handle, b: T.handle):
+        A = T.match_buffer(a, (16,), "float32")
+        B = T.match_buffer(b, (16,), "float32")
+        T.func_attr({"global_symbol": "main", "tir.noalias": True})
+        for i_0 in T.serial(T.ceildiv(16, 4 * T.vscale())):
+            for i_1 in T.vectorized(4 * T.vscale()):
+                if i_0 * 4 * T.vscale() + i_1 < 14:
+                    B[i_0 * 4 * T.vscale() + i_1] = A[i_0 * 4 * T.vscale() + i_1] + 1.0
+
+    with tvm.target.Target(target):

Review Comment:
   This style of(this was created for legacy usage), instead, attach the target attribute to the PrimFunc itself



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

Posted by "lhutton1 (via GitHub)" <gi...@apache.org>.
lhutton1 commented on code in PR #16966:
URL: https://github.com/apache/tvm/pull/16966#discussion_r1599992854


##########
src/tir/transforms/vectorize_loop.cc:
##########
@@ -72,6 +72,126 @@ inline PrimExpr BroadcastTo(PrimExpr e, int lanes, bool is_scalable) {
   return Broadcast(e, CreateNewLanes(is_scalable, lanes));
 }
 
+bool EnableBufferLevelPredication() {

Review Comment:
   Thanks for the detailed description. To save the number of changes in this PR getting even larger, I've added the minimum changes necessary for the vectorizer pass. Perhaps we can clean this up and fix the target checks in the analyzer in a separate PR?
   
   Edit: on second thoughts, I'm not sure how the analyzer should behave when the target has been overridden. Should the analyzer constructor now accept a `Target` parameter which can be passed on to the sub analyzers that require it? Propagating `Target` correctly from all passes that make use of the Analyzer seems like quite a big task.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

Posted by "lhutton1 (via GitHub)" <gi...@apache.org>.
lhutton1 commented on code in PR #16966:
URL: https://github.com/apache/tvm/pull/16966#discussion_r1599975814


##########
python/tvm/tir/buffer.py:
##########
@@ -141,6 +141,57 @@ def vstore(self, begin, value):
         begin = (begin,) if isinstance(begin, (int, PrimExpr)) else begin
         return _ffi_api.BufferVStore(self, begin, value)  # type: ignore
 
+    def load(self, indices, predicate=None):

Review Comment:
   Got it, thanks. I think part of the confusion here was that I wasn't sure whether it was intended to be used this way - I couldn't find vload/vstore used this way in the codebase. I've amended to use vload/vstore.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

Posted by "leandron (via GitHub)" <gi...@apache.org>.
leandron merged PR #16966:
URL: https://github.com/apache/tvm/pull/16966


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

Posted by "Lunderberg (via GitHub)" <gi...@apache.org>.
Lunderberg commented on code in PR #16966:
URL: https://github.com/apache/tvm/pull/16966#discussion_r1598715370


##########
src/tir/transforms/vectorize_loop.cc:
##########
@@ -72,6 +72,126 @@ inline PrimExpr BroadcastTo(PrimExpr e, int lanes, bool is_scalable) {
   return Broadcast(e, CreateNewLanes(is_scalable, lanes));
 }
 
+bool EnableBufferLevelPredication() {

Review Comment:
   To my knowledge, there isn't a general utility that would provide it, though I definitely agree on its usefulness.
   
   There's currently a couple similar utilities, such as [`IRMutatorWithAnalyzer`](https://github.com/apache/tvm/blob/main/src/arith/ir_mutator_with_analyzer.h#L45) and [`IRVisitorWithAnalyzer`](https://github.com/apache/tvm/blob/main/src/arith/ir_visitor_with_analyzer.h#L35), which allow context-dependent algebraic simplifications. The easiest way to implement such a utility for target-aware simplifications would be to extract out the context-dependent target tracking from an existing visitor/mutator (e.g. [here](https://github.com/apache/tvm/blob/main/src/tir/transforms/lower_device_kernel_launch.cc#L65), used in the `LowerDeviceKernelLaunch` transform) into a `IRVisitorWithTarget` and `IRMutatorWithTarget`, which is then subclassed again by each transform that requires it.
   
   For long-term maintainability, I like your suggestion of mixins much more than the subclasses, because they could be composed together much more easily.  Implemented with subclasses, a hypothetical transform that requires both the context-dependent target and context-dependent algebraic simplifications would have incorrect behavior from multiple inheritance, and would need to re-implement one or the other functionalities.
   
   Unfortunately, I don't think the current `tir::StmtVisitor` API could be used to support mixins, either through multiple inheritance or through composition.  The main issue is that the implementable virtual functions (e.g. `tir::StmtVisitor::VisitStmt_(const tir::ForNode*)`) aren't granular enough.  In order to compose separate transforms that provide context-dependent information, it would require separate virtual functions for pre-visit and post-visit.  I've toyed around with different variations of the TIR stmt/expr visitor/mutator implementations, but haven't had the time to make anything PR-able.
   
   TL;DR: It doesn't currently exist, would be very useful to have, and a subclass would be sufficient for this PR, though long-term maintainability concerns may appear at some point in the future.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

Posted by "Lunderberg (via GitHub)" <gi...@apache.org>.
Lunderberg commented on code in PR #16966:
URL: https://github.com/apache/tvm/pull/16966#discussion_r1598726836


##########
src/target/llvm/codegen_llvm.cc:
##########
@@ -1768,11 +1774,17 @@ llvm::Value* CodeGenLLVM::VisitExpr_(const BufferLoadNode* op) {
 
   std::vector<llvm::Value*> loads;
 
-  auto make_load = [this, &loads](TypedPointer buffer_ptr, int /* subelement_i */, int alignment,
-                                  bool is_volatile) {
+  auto make_load = [this, &loads](TypedPointer buffer_ptr, int /* subelement_i */,
+                                  llvm::Value* predicate, int alignment, bool is_volatile) {
 #if TVM_LLVM_VERSION >= 110
-    auto load = builder_->CreateAlignedLoad(buffer_ptr.type, buffer_ptr.addr,
-                                            llvm::Align(alignment), is_volatile);
+    llvm::Instruction* load = nullptr;
+    if (predicate != NULL) {
+      load = builder_->CreateMaskedLoad(buffer_ptr.type, buffer_ptr.addr, llvm::Align(alignment),
+                                        predicate);

Review Comment:
   Thank you, and throwing the exception makes sense.  (And we should probably also add a unit test to verify that the exception is thrown.)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

Posted by "lhutton1 (via GitHub)" <gi...@apache.org>.
lhutton1 commented on code in PR #16966:
URL: https://github.com/apache/tvm/pull/16966#discussion_r1598722254


##########
src/target/llvm/codegen_llvm.cc:
##########
@@ -1768,11 +1774,17 @@ llvm::Value* CodeGenLLVM::VisitExpr_(const BufferLoadNode* op) {
 
   std::vector<llvm::Value*> loads;
 
-  auto make_load = [this, &loads](TypedPointer buffer_ptr, int /* subelement_i */, int alignment,
-                                  bool is_volatile) {
+  auto make_load = [this, &loads](TypedPointer buffer_ptr, int /* subelement_i */,
+                                  llvm::Value* predicate, int alignment, bool is_volatile) {
 #if TVM_LLVM_VERSION >= 110
-    auto load = builder_->CreateAlignedLoad(buffer_ptr.type, buffer_ptr.addr,
-                                            llvm::Align(alignment), is_volatile);
+    llvm::Instruction* load = nullptr;
+    if (predicate != NULL) {
+      load = builder_->CreateMaskedLoad(buffer_ptr.type, buffer_ptr.addr, llvm::Align(alignment),
+                                        predicate);
+    } else {
+      load = builder_->CreateAlignedLoad(buffer_ptr.type, buffer_ptr.addr, llvm::Align(alignment),
+                                         is_volatile);
+    }
 #elif TVM_LLVM_VERSION >= 80
     auto load =

Review Comment:
   Agreed, thanks. I've added support for previous versions of LLVM. I've checked the build with the following versions of LLVM: 7*, 8*, 9*, 10, 11, 12, 13, 17
   
   \* fails to build due to other seemingly unrelated errors



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

Posted by "Lunderberg (via GitHub)" <gi...@apache.org>.
Lunderberg commented on PR #16966:
URL: https://github.com/apache/tvm/pull/16966#issuecomment-2137560131

   Apologies for not getting back to the review, and thank you for making the changes!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

Posted by "lhutton1 (via GitHub)" <gi...@apache.org>.
lhutton1 commented on code in PR #16966:
URL: https://github.com/apache/tvm/pull/16966#discussion_r1599972721


##########
include/tvm/script/ir_builder/tir/ir.h:
##########
@@ -411,8 +411,10 @@ Var EnvThread(String thread_tag, DataType dtype = DataType::Int(32));
  * \param buffer The buffer.
  * \param value The value to be stored.
  * \param indices The indices location to be stored.
+ * \param predicate A vector mask of int1 values indicating which lanes of a vector are to be

Review Comment:
   Done



##########
python/tvm/tir/expr.py:
##########
@@ -1093,20 +1093,27 @@ class BufferLoad(PrimExprWithOp):
         The buffer to be loaded.
 
     indices : List[PrimExpr]
-        The buffer indices.
+        The buffer indices to load values from.
 
     span : Optional[Span]
         The location of this expression in the source code.
+
+    predicate : Optional[PrimExpr]
+        A vector mask of int1 values indicating which lanes of a vector are to be loaded.

Review Comment:
   Done. There was a bit of confusion here since LLVM intrinsics expect predicate masks of `int1` values. However, boolean types are already converted to `int1` in the codegen backend. So I have required `uint1`/boolean types in TVM.



##########
include/tvm/tir/expr.h:
##########
@@ -675,7 +679,8 @@ class BufferLoadNode : public PrimExprNode {
  */
 class BufferLoad : public PrimExpr {
  public:
-  TVM_DLL explicit BufferLoad(Buffer buffer, Array<PrimExpr> indices, Span span = Span());
+  TVM_DLL explicit BufferLoad(Buffer buffer, Array<PrimExpr> indices,
+                              PrimExpr predicate = PrimExpr(), Span span = Span());

Review Comment:
   Done



##########
src/tir/transforms/vectorize_loop.cc:
##########
@@ -72,6 +72,126 @@ inline PrimExpr BroadcastTo(PrimExpr e, int lanes, bool is_scalable) {
   return Broadcast(e, CreateNewLanes(is_scalable, lanes));
 }
 
+bool EnableBufferLevelPredication() {

Review Comment:
   Thanks for the detailed description. To save this PR exploding further, I've added the minimum changes necessary for the vectorizer pass. Perhaps we can clean this up and fix the target checks in the analyzer in a separate PR?



##########
python/tvm/tir/buffer.py:
##########
@@ -141,6 +141,57 @@ def vstore(self, begin, value):
         begin = (begin,) if isinstance(begin, (int, PrimExpr)) else begin
         return _ffi_api.BufferVStore(self, begin, value)  # type: ignore
 
+    def load(self, indices, predicate=None):

Review Comment:
   Got it, thanks. I think part of the confusion here was that I wasn't sure whether it was intended to be used this way - I couldn't find vload/vstore used this way in the codebase



##########
tests/python/tir-transform/test_tir_transform_vectorize.py:
##########
@@ -488,5 +529,170 @@ def main(A: T.Buffer((16,), "float32")):
             tvm.tir.transform.VectorizeLoop()(Mod)
 
 
+def test_vectorize_and_predicate_all_buffer_loads_stores():
+    @T.prim_func
+    def before(a: T.handle, b: T.handle):
+        A = T.match_buffer(a, (16,), "float32")
+        B = T.match_buffer(b, (16,), "float32")
+        T.func_attr({"global_symbol": "main", "tir.noalias": True})
+        for i_0 in T.serial(T.ceildiv(14, 4)):
+            for i_1 in T.vectorized(4):
+                if i_0 * 4 + i_1 < 14:
+                    B[i_0 * 4 + i_1] = A[i_0 * 4 + i_1] + 1.0
+
+    @T.prim_func
+    def expected(a: T.handle, b: T.handle):
+        A = T.match_buffer(a, (16,), "float32")
+        B = T.match_buffer(b, (16,), "float32")
+        T.func_attr({"global_symbol": "main", "tir.noalias": T.bool(True)})
+        for i_0 in range(4):
+            load_a = T.meta_var(
+                A.load(
+                    [T.Ramp(i_0 * 4, 1, 4)], predicate=T.get_active_lane_mask("int1x4", i_0 * 4, 14)
+                )
+            )
+            add_1 = T.meta_var(load_a + T.Broadcast(T.float32(1), 4))
+            B.store(
+                add_1,
+                [T.Ramp(i_0 * 4, 1, 4)],
+                predicate=T.get_active_lane_mask("int1x4", i_0 * 4, 14),
+            )
+
+    mod = tvm.IRModule.from_expr(before)
+    with tvm.transform.PassContext(config={"tir.enable_buffer_level_predication": True}):
+        after = tvm.tir.transform.VectorizeLoop()(mod)["main"]
+    tvm.ir.assert_structural_equal(after, expected)
+
+
+def test_vectorize_and_predicate_some_buffer_loads_stores():
+    # Currently revert to scalarizing the block if not all accesses
+    # have been predicated, otherwise incorrect code is generated.
+    @T.prim_func
+    def before(a: T.handle, b: T.handle):
+        A = T.match_buffer(a, (16,), "float32")
+        B = T.match_buffer(b, (16,), "float32")
+        T.func_attr({"global_symbol": "main", "tir.noalias": True})
+        for i_0 in T.serial(T.ceildiv(14, 4)):
+            for i_1 in T.vectorized(4):
+                if i_0 * 4 + i_1 < 14:
+                    B[i_0 * 4 + i_1] = A[i_0] + 1.0
+
+    @T.prim_func
+    def expected(a: T.handle, b: T.handle):
+        A = T.match_buffer(a, (16,), "float32")
+        B = T.match_buffer(b, (16,), "float32")
+        T.func_attr({"global_symbol": "main", "tir.noalias": T.bool(True)})
+        for i_0, i_1_s in T.grid(4, 4):
+            if i_0 * 4 + i_1_s < 14:
+                B[i_0 * 4 + i_1_s] = A[i_0] + T.float32(1)
+
+    mod = tvm.IRModule.from_expr(before)
+    with tvm.transform.PassContext(config={"tir.enable_buffer_level_predication": True}):
+        after = tvm.tir.transform.VectorizeLoop()(mod)["main"]
+    tvm.ir.assert_structural_equal(after, expected)
+
+
+def test_vectorize_and_predicate_multiple_access_statements():
+    @T.prim_func
+    def before(a: T.handle, b: T.handle):
+        A = T.match_buffer(a, (16,), "float32")
+        B = T.match_buffer(b, (16,), "float32")
+        T.func_attr({"global_symbol": "main", "tir.noalias": True})
+        for i_0 in T.serial(T.ceildiv(14, 4)):
+            for i_1 in T.vectorized(4):
+                if i_0 * 4 + i_1 < 14:
+                    A[i_0 * 4 + i_1] = 2.0
+                    B[i_0 * 4 + i_1] = 1.0
+
+    @T.prim_func
+    def expected(a: T.handle, b: T.handle):
+        A = T.match_buffer(a, (16,), "float32")
+        B = T.match_buffer(b, (16,), "float32")
+        T.func_attr({"global_symbol": "main", "tir.noalias": T.bool(True)})
+        for i_0 in range(4):
+            A.store(
+                T.Broadcast(T.float32(2), 4),
+                [T.Ramp(i_0 * 4, 1, 4)],
+                predicate=T.get_active_lane_mask("int1x4", i_0 * 4, 14),
+            )
+            B.store(
+                T.Broadcast(T.float32(1), 4),
+                [T.Ramp(i_0 * 4, 1, 4)],
+                predicate=T.get_active_lane_mask("int1x4", i_0 * 4, 14),
+            )
+
+    before_mod = tvm.IRModule.from_expr(before)
+    with tvm.transform.PassContext(config={"tir.enable_buffer_level_predication": True}):
+        after = tvm.tir.transform.VectorizeLoop()(before_mod)["main"]
+    tvm.ir.assert_structural_equal(after, expected)
+
+
+def test_vectorize_and_predicate_invalid_conditions():
+    @T.prim_func
+    def before(a: T.handle, b: T.handle):
+        A = T.match_buffer(a, (16,), "float32")
+        B = T.match_buffer(b, (16,), "float32")
+        T.func_attr({"global_symbol": "main", "tir.noalias": True})
+        for i_0 in T.serial(T.ceildiv(14, 4)):
+            for i_1 in T.vectorized(4):
+                if i_0 * 4 + i_1 > 14:
+                    A[i_0 * 4 + i_1] = 2.0
+                if 14 < i_0 * 4 + i_1:
+                    A[i_0 * 4 + i_1] = 2.0
+                if i_0 * 4 + i_1 < i_0 * 4 + i_1:
+                    A[i_0 * 4 + i_1] = 2.0
+
+    @T.prim_func
+    def expected(a: T.handle, b: T.handle):
+        A = T.match_buffer(a, (16,), "float32")
+        B = T.match_buffer(b, (16,), "float32")
+        T.func_attr({"global_symbol": "main", "tir.noalias": T.bool(True)})
+        for i_0 in range(4):
+            for i_1_s in range(4):
+                if i_0 * 4 + i_1_s > 14:
+                    A[i_0 * 4 + i_1_s] = T.float32(2)
+            for i_1_s in range(4):
+                if 14 < i_0 * 4 + i_1_s:
+                    A[i_0 * 4 + i_1_s] = T.float32(2)
+            for i_1_s in range(4):
+                if i_0 * 4 + i_1_s < i_0 * 4 + i_1_s:
+                    A[i_0 * 4 + i_1_s] = T.float32(2)
+
+    before_mod = tvm.IRModule.from_expr(before)
+    with tvm.transform.PassContext(config={"tir.enable_buffer_level_predication": True}):
+        after = tvm.tir.transform.VectorizeLoop()(before_mod)["main"]
+    tvm.ir.assert_structural_equal(after, expected)
+
+
+def test_vectorize_with_explicitly_disabled_buffer_level_predication():
+    # Since the target is has the SVe feature, buffer level predication is enabled
+    # by default. However, it has been explicitely disabled by the pass context
+    # option, so no buffer-level predicates should be added.
+    @T.prim_func
+    def before(a: T.handle, b: T.handle):
+        A = T.match_buffer(a, (16,), "float32")
+        B = T.match_buffer(b, (16,), "float32")
+        T.func_attr({"global_symbol": "main", "tir.noalias": True})
+        for i_0 in T.serial(T.ceildiv(14, 4)):
+            for i_1 in T.vectorized(4):
+                if i_0 * 4 + i_1 < 14:
+                    B[i_0 * 4 + i_1] = A[i_0 * 4 + i_1] + 1.0
+
+    @T.prim_func
+    def expected(a: T.handle, b: T.handle):
+        A = T.match_buffer(a, (16,), "float32")
+        B = T.match_buffer(b, (16,), "float32")
+        T.func_attr({"global_symbol": "main", "tir.noalias": True})
+        for i_0, i_1_s in T.grid(4, 4):
+            if i_0 * 4 + i_1_s < 14:
+                B[i_0 * 4 + i_1_s] = A[i_0 * 4 + i_1_s] + T.float32(1)
+
+    mod = tvm.IRModule.from_expr(before)
+    with tvm.transform.PassContext(config={"tir.enable_buffer_level_predication": False}):
+        with tvm.target.Target("llvm -mtriple=aarch64-linux-gnu -mattr=+sve"):
+            after = tvm.tir.transform.VectorizeLoop()(mod)["main"]
+    tvm.ir.assert_structural_equal(after, expected)
+

Review Comment:
   Done



##########
src/target/llvm/codegen_llvm.cc:
##########
@@ -1768,11 +1774,17 @@ llvm::Value* CodeGenLLVM::VisitExpr_(const BufferLoadNode* op) {
 
   std::vector<llvm::Value*> loads;
 
-  auto make_load = [this, &loads](TypedPointer buffer_ptr, int /* subelement_i */, int alignment,
-                                  bool is_volatile) {
+  auto make_load = [this, &loads](TypedPointer buffer_ptr, int /* subelement_i */,
+                                  llvm::Value* predicate, int alignment, bool is_volatile) {
 #if TVM_LLVM_VERSION >= 110
-    auto load = builder_->CreateAlignedLoad(buffer_ptr.type, buffer_ptr.addr,
-                                            llvm::Align(alignment), is_volatile);
+    llvm::Instruction* load = nullptr;
+    if (predicate != NULL) {
+      load = builder_->CreateMaskedLoad(buffer_ptr.type, buffer_ptr.addr, llvm::Align(alignment),
+                                        predicate);

Review Comment:
   Done



##########
src/target/llvm/codegen_llvm.cc:
##########
@@ -1768,11 +1774,17 @@ llvm::Value* CodeGenLLVM::VisitExpr_(const BufferLoadNode* op) {
 
   std::vector<llvm::Value*> loads;
 
-  auto make_load = [this, &loads](TypedPointer buffer_ptr, int /* subelement_i */, int alignment,
-                                  bool is_volatile) {
+  auto make_load = [this, &loads](TypedPointer buffer_ptr, int /* subelement_i */,
+                                  llvm::Value* predicate, int alignment, bool is_volatile) {
 #if TVM_LLVM_VERSION >= 110
-    auto load = builder_->CreateAlignedLoad(buffer_ptr.type, buffer_ptr.addr,
-                                            llvm::Align(alignment), is_volatile);
+    llvm::Instruction* load = nullptr;
+    if (predicate != NULL) {
+      load = builder_->CreateMaskedLoad(buffer_ptr.type, buffer_ptr.addr, llvm::Align(alignment),
+                                        predicate);
+    } else {
+      load = builder_->CreateAlignedLoad(buffer_ptr.type, buffer_ptr.addr, llvm::Align(alignment),
+                                         is_volatile);
+    }
 #elif TVM_LLVM_VERSION >= 80
     auto load =

Review Comment:
   Agreed, thanks. I've added support for previous versions of LLVM. I've checked the build with the following versions of LLVM: 7*, 8*, 9*, 10, 11, 12, 13, 17
   
   * fails to build due to other seemingly unrelated errors



##########
tests/python/codegen/test_target_codegen_aarch64.py:
##########
@@ -700,5 +700,31 @@ def before(a: T.handle):
     assert "get.active.lane.mask" in ll
 
 
+@pytest.mark.skipif(

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

Posted by "lhutton1 (via GitHub)" <gi...@apache.org>.
lhutton1 commented on code in PR #16966:
URL: https://github.com/apache/tvm/pull/16966#discussion_r1599992854


##########
src/tir/transforms/vectorize_loop.cc:
##########
@@ -72,6 +72,126 @@ inline PrimExpr BroadcastTo(PrimExpr e, int lanes, bool is_scalable) {
   return Broadcast(e, CreateNewLanes(is_scalable, lanes));
 }
 
+bool EnableBufferLevelPredication() {

Review Comment:
   Thanks for the detailed description. To save the number of changes in this PR getting even larger, I've added the minimum changes necessary for the vectorizer pass. Perhaps we can clean this up and fix the target checks in the analyzer in a separate PR?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

Posted by "lhutton1 (via GitHub)" <gi...@apache.org>.
lhutton1 commented on code in PR #16966:
URL: https://github.com/apache/tvm/pull/16966#discussion_r1606010405


##########
src/tir/ir/stmt.cc:
##########
@@ -476,29 +477,39 @@ BufferStore::BufferStore(Buffer buffer, PrimExpr value, Array<PrimExpr> indices,
   ICHECK(!(is_index_scalable && is_buffer_dtype_scalable))
       << "Index dtype and buffer dtype can't both be scalable.";
 
-  if (is_index_scalable || is_buffer_dtype_scalable) {
-    ICHECK(is_value_dtype_scalable) << "Can't store non-scalable data into scalable buffer";
+  if (predicate.defined()) {
+    bool is_predicate_dtype_scalable = predicate.value().dtype().is_scalable_vector();
+    ICHECK_EQ(is_value_dtype_scalable, is_predicate_dtype_scalable)
+        << "Predicate mask dtype and value dtype must both be scalable.";
   }
 
-  int index_lanes;
-  if (indices.empty()) {
-    index_lanes = 1;
-  } else if (is_index_scalable) {
-    index_lanes = indices.back().dtype().vscale_factor();
-  } else {
-    index_lanes = indices.back().dtype().lanes();
+  if (is_index_scalable || is_buffer_dtype_scalable) {
+    ICHECK(is_value_dtype_scalable) << "Can't store non-scalable data into scalable buffer";
   }
 
-  int buffer_lanes =
-      is_buffer_dtype_scalable ? buffer->dtype.vscale_factor() : buffer->dtype.lanes();
-  int value_dtype_lanes =
-      is_value_dtype_scalable ? value.dtype().vscale_factor() : value.dtype().lanes();
+  int index_lanes = indices.empty() ? 1 : indices.back().dtype().get_lanes_or_vscale_factor();
+  int buffer_lanes = buffer->dtype.get_lanes_or_vscale_factor();
+  int value_dtype_lanes = value.dtype().get_lanes_or_vscale_factor();
 
   ICHECK_EQ(index_lanes * buffer_lanes, value_dtype_lanes)
       << "Cannot store value with " << value_dtype_lanes << ", expected value with "
       << index_lanes * buffer_lanes << " (" << index_lanes << " index lanes * " << buffer_lanes
       << " buffer element lanes)";
 
+  if (predicate.defined()) {
+    DataType predicate_dtype = predicate.value().dtype();
+    int predicate_dtype_lanes = predicate_dtype.get_lanes_or_vscale_factor();
+    ICHECK_EQ(value_dtype_lanes, predicate_dtype_lanes)

Review Comment:
   Good catch, yes we should



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

Posted by "ekalda (via GitHub)" <gi...@apache.org>.
ekalda commented on code in PR #16966:
URL: https://github.com/apache/tvm/pull/16966#discussion_r1605010649


##########
include/tvm/tir/buffer.h:
##########
@@ -209,14 +209,20 @@ class Buffer : public ObjectRef {
    * \brief Create an Expr that does a vector load at begin index.
    * \param begin The beginning index
    * \param dtype The data type to be loaded.
+   * \param predicate A vector mask of boolean values indicating which lanes of a vector are to be
+   * stored. The number lanes of the mask must be equal to the number of lanes in value.

Review Comment:
   Nit: "storing lanes of a vector" is not relevant to a loading API documentation, also there is no `value` argument here, so it's a bit ambiguous. (Same for other documentation of loading in other places)



##########
src/tir/ir/stmt.cc:
##########
@@ -476,29 +477,39 @@ BufferStore::BufferStore(Buffer buffer, PrimExpr value, Array<PrimExpr> indices,
   ICHECK(!(is_index_scalable && is_buffer_dtype_scalable))
       << "Index dtype and buffer dtype can't both be scalable.";
 
-  if (is_index_scalable || is_buffer_dtype_scalable) {
-    ICHECK(is_value_dtype_scalable) << "Can't store non-scalable data into scalable buffer";
+  if (predicate.defined()) {
+    bool is_predicate_dtype_scalable = predicate.value().dtype().is_scalable_vector();
+    ICHECK_EQ(is_value_dtype_scalable, is_predicate_dtype_scalable)
+        << "Predicate mask dtype and value dtype must both be scalable.";
   }
 
-  int index_lanes;
-  if (indices.empty()) {
-    index_lanes = 1;
-  } else if (is_index_scalable) {
-    index_lanes = indices.back().dtype().vscale_factor();
-  } else {
-    index_lanes = indices.back().dtype().lanes();
+  if (is_index_scalable || is_buffer_dtype_scalable) {
+    ICHECK(is_value_dtype_scalable) << "Can't store non-scalable data into scalable buffer";
   }
 
-  int buffer_lanes =
-      is_buffer_dtype_scalable ? buffer->dtype.vscale_factor() : buffer->dtype.lanes();
-  int value_dtype_lanes =
-      is_value_dtype_scalable ? value.dtype().vscale_factor() : value.dtype().lanes();
+  int index_lanes = indices.empty() ? 1 : indices.back().dtype().get_lanes_or_vscale_factor();
+  int buffer_lanes = buffer->dtype.get_lanes_or_vscale_factor();
+  int value_dtype_lanes = value.dtype().get_lanes_or_vscale_factor();
 
   ICHECK_EQ(index_lanes * buffer_lanes, value_dtype_lanes)
       << "Cannot store value with " << value_dtype_lanes << ", expected value with "
       << index_lanes * buffer_lanes << " (" << index_lanes << " index lanes * " << buffer_lanes
       << " buffer element lanes)";
 
+  if (predicate.defined()) {
+    DataType predicate_dtype = predicate.value().dtype();
+    int predicate_dtype_lanes = predicate_dtype.get_lanes_or_vscale_factor();
+    ICHECK_EQ(value_dtype_lanes, predicate_dtype_lanes)

Review Comment:
   Should we have a similar check in the `BufferLoad` node? 



##########
src/tir/transforms/inject_rolling_buffer.cc:
##########
@@ -257,7 +257,9 @@ class RollingBufferInjector : public StmtExprMutator {
           indices.push_back(index);
         }
       }
-      Stmt buffer_store = BufferStore(op->buffer, op->value, indices, op->span);
+      ICHECK(!op->predicate.defined())
+          << "Predicated buffer store is not current supported in the inject rolling buffer pass.";

Review Comment:
   Nit:
   
   ```suggestion
             << "Predicated buffer store is not currently supported in the inject rolling buffer pass.";
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

Posted by "tqchen (via GitHub)" <gi...@apache.org>.
tqchen commented on code in PR #16966:
URL: https://github.com/apache/tvm/pull/16966#discussion_r1618046951


##########
tests/python/codegen/test_target_codegen_aarch64.py:
##########
@@ -780,5 +780,31 @@ def before(a: T.handle):
     assert "get.active.lane.mask" in ll
 
 
+@pytest.mark.skipif(
+    llvm_version_major() < 11,
+    reason="Vscale and get.active.lane.mask are not supported in earlier versions of LLVM",
+)
+def test_predicated_scalable_buffer():
+    target = "llvm -mtriple=aarch64-linux-gnu -mattr=+sve"
+
+    @T.prim_func
+    def before(a: T.handle, b: T.handle):
+        A = T.match_buffer(a, (16,), "float32")
+        B = T.match_buffer(b, (16,), "float32")
+        T.func_attr({"global_symbol": "main", "tir.noalias": True})
+        for i_0 in T.serial(T.ceildiv(16, 4 * T.vscale())):
+            for i_1 in T.vectorized(4 * T.vscale()):
+                if i_0 * 4 * T.vscale() + i_1 < 14:
+                    B[i_0 * 4 * T.vscale() + i_1] = A[i_0 * 4 * T.vscale() + i_1] + 1.0
+
+    with tvm.target.Target(target):

Review Comment:
   This style moving foward, instead, attach the target attribute to the PrimFunc itself



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

Posted by "Lunderberg (via GitHub)" <gi...@apache.org>.
Lunderberg commented on code in PR #16966:
URL: https://github.com/apache/tvm/pull/16966#discussion_r1594285519


##########
include/tvm/script/ir_builder/tir/ir.h:
##########
@@ -411,8 +411,10 @@ Var EnvThread(String thread_tag, DataType dtype = DataType::Int(32));
  * \param buffer The buffer.
  * \param value The value to be stored.
  * \param indices The indices location to be stored.
+ * \param predicate A vector mask of int1 values indicating which lanes of a vector are to be

Review Comment:
   Can we specify that the number of lanes for the `predicate` must be equal to the number of lanes in the `value`?



##########
tests/python/tir-transform/test_tir_transform_vectorize.py:
##########
@@ -488,5 +529,170 @@ def main(A: T.Buffer((16,), "float32")):
             tvm.tir.transform.VectorizeLoop()(Mod)
 
 
+def test_vectorize_and_predicate_all_buffer_loads_stores():
+    @T.prim_func
+    def before(a: T.handle, b: T.handle):
+        A = T.match_buffer(a, (16,), "float32")
+        B = T.match_buffer(b, (16,), "float32")
+        T.func_attr({"global_symbol": "main", "tir.noalias": True})
+        for i_0 in T.serial(T.ceildiv(14, 4)):
+            for i_1 in T.vectorized(4):
+                if i_0 * 4 + i_1 < 14:
+                    B[i_0 * 4 + i_1] = A[i_0 * 4 + i_1] + 1.0
+
+    @T.prim_func
+    def expected(a: T.handle, b: T.handle):
+        A = T.match_buffer(a, (16,), "float32")
+        B = T.match_buffer(b, (16,), "float32")
+        T.func_attr({"global_symbol": "main", "tir.noalias": T.bool(True)})
+        for i_0 in range(4):
+            load_a = T.meta_var(
+                A.load(
+                    [T.Ramp(i_0 * 4, 1, 4)], predicate=T.get_active_lane_mask("int1x4", i_0 * 4, 14)
+                )
+            )
+            add_1 = T.meta_var(load_a + T.Broadcast(T.float32(1), 4))
+            B.store(
+                add_1,
+                [T.Ramp(i_0 * 4, 1, 4)],
+                predicate=T.get_active_lane_mask("int1x4", i_0 * 4, 14),
+            )
+
+    mod = tvm.IRModule.from_expr(before)
+    with tvm.transform.PassContext(config={"tir.enable_buffer_level_predication": True}):
+        after = tvm.tir.transform.VectorizeLoop()(mod)["main"]
+    tvm.ir.assert_structural_equal(after, expected)
+
+
+def test_vectorize_and_predicate_some_buffer_loads_stores():
+    # Currently revert to scalarizing the block if not all accesses
+    # have been predicated, otherwise incorrect code is generated.
+    @T.prim_func
+    def before(a: T.handle, b: T.handle):
+        A = T.match_buffer(a, (16,), "float32")
+        B = T.match_buffer(b, (16,), "float32")
+        T.func_attr({"global_symbol": "main", "tir.noalias": True})
+        for i_0 in T.serial(T.ceildiv(14, 4)):
+            for i_1 in T.vectorized(4):
+                if i_0 * 4 + i_1 < 14:
+                    B[i_0 * 4 + i_1] = A[i_0] + 1.0
+
+    @T.prim_func
+    def expected(a: T.handle, b: T.handle):
+        A = T.match_buffer(a, (16,), "float32")
+        B = T.match_buffer(b, (16,), "float32")
+        T.func_attr({"global_symbol": "main", "tir.noalias": T.bool(True)})
+        for i_0, i_1_s in T.grid(4, 4):
+            if i_0 * 4 + i_1_s < 14:
+                B[i_0 * 4 + i_1_s] = A[i_0] + T.float32(1)
+
+    mod = tvm.IRModule.from_expr(before)
+    with tvm.transform.PassContext(config={"tir.enable_buffer_level_predication": True}):
+        after = tvm.tir.transform.VectorizeLoop()(mod)["main"]
+    tvm.ir.assert_structural_equal(after, expected)
+
+
+def test_vectorize_and_predicate_multiple_access_statements():
+    @T.prim_func
+    def before(a: T.handle, b: T.handle):
+        A = T.match_buffer(a, (16,), "float32")
+        B = T.match_buffer(b, (16,), "float32")
+        T.func_attr({"global_symbol": "main", "tir.noalias": True})
+        for i_0 in T.serial(T.ceildiv(14, 4)):
+            for i_1 in T.vectorized(4):
+                if i_0 * 4 + i_1 < 14:
+                    A[i_0 * 4 + i_1] = 2.0
+                    B[i_0 * 4 + i_1] = 1.0
+
+    @T.prim_func
+    def expected(a: T.handle, b: T.handle):
+        A = T.match_buffer(a, (16,), "float32")
+        B = T.match_buffer(b, (16,), "float32")
+        T.func_attr({"global_symbol": "main", "tir.noalias": T.bool(True)})
+        for i_0 in range(4):
+            A.store(
+                T.Broadcast(T.float32(2), 4),
+                [T.Ramp(i_0 * 4, 1, 4)],
+                predicate=T.get_active_lane_mask("int1x4", i_0 * 4, 14),
+            )
+            B.store(
+                T.Broadcast(T.float32(1), 4),
+                [T.Ramp(i_0 * 4, 1, 4)],
+                predicate=T.get_active_lane_mask("int1x4", i_0 * 4, 14),
+            )
+
+    before_mod = tvm.IRModule.from_expr(before)
+    with tvm.transform.PassContext(config={"tir.enable_buffer_level_predication": True}):
+        after = tvm.tir.transform.VectorizeLoop()(before_mod)["main"]
+    tvm.ir.assert_structural_equal(after, expected)
+
+
+def test_vectorize_and_predicate_invalid_conditions():
+    @T.prim_func
+    def before(a: T.handle, b: T.handle):
+        A = T.match_buffer(a, (16,), "float32")
+        B = T.match_buffer(b, (16,), "float32")
+        T.func_attr({"global_symbol": "main", "tir.noalias": True})
+        for i_0 in T.serial(T.ceildiv(14, 4)):
+            for i_1 in T.vectorized(4):
+                if i_0 * 4 + i_1 > 14:
+                    A[i_0 * 4 + i_1] = 2.0
+                if 14 < i_0 * 4 + i_1:
+                    A[i_0 * 4 + i_1] = 2.0
+                if i_0 * 4 + i_1 < i_0 * 4 + i_1:
+                    A[i_0 * 4 + i_1] = 2.0
+
+    @T.prim_func
+    def expected(a: T.handle, b: T.handle):
+        A = T.match_buffer(a, (16,), "float32")
+        B = T.match_buffer(b, (16,), "float32")
+        T.func_attr({"global_symbol": "main", "tir.noalias": T.bool(True)})
+        for i_0 in range(4):
+            for i_1_s in range(4):
+                if i_0 * 4 + i_1_s > 14:
+                    A[i_0 * 4 + i_1_s] = T.float32(2)
+            for i_1_s in range(4):
+                if 14 < i_0 * 4 + i_1_s:
+                    A[i_0 * 4 + i_1_s] = T.float32(2)
+            for i_1_s in range(4):
+                if i_0 * 4 + i_1_s < i_0 * 4 + i_1_s:
+                    A[i_0 * 4 + i_1_s] = T.float32(2)
+
+    before_mod = tvm.IRModule.from_expr(before)
+    with tvm.transform.PassContext(config={"tir.enable_buffer_level_predication": True}):
+        after = tvm.tir.transform.VectorizeLoop()(before_mod)["main"]
+    tvm.ir.assert_structural_equal(after, expected)
+
+
+def test_vectorize_with_explicitly_disabled_buffer_level_predication():
+    # Since the target is has the SVe feature, buffer level predication is enabled
+    # by default. However, it has been explicitely disabled by the pass context
+    # option, so no buffer-level predicates should be added.
+    @T.prim_func
+    def before(a: T.handle, b: T.handle):
+        A = T.match_buffer(a, (16,), "float32")
+        B = T.match_buffer(b, (16,), "float32")
+        T.func_attr({"global_symbol": "main", "tir.noalias": True})
+        for i_0 in T.serial(T.ceildiv(14, 4)):
+            for i_1 in T.vectorized(4):
+                if i_0 * 4 + i_1 < 14:
+                    B[i_0 * 4 + i_1] = A[i_0 * 4 + i_1] + 1.0
+
+    @T.prim_func
+    def expected(a: T.handle, b: T.handle):
+        A = T.match_buffer(a, (16,), "float32")
+        B = T.match_buffer(b, (16,), "float32")
+        T.func_attr({"global_symbol": "main", "tir.noalias": True})
+        for i_0, i_1_s in T.grid(4, 4):
+            if i_0 * 4 + i_1_s < 14:
+                B[i_0 * 4 + i_1_s] = A[i_0 * 4 + i_1_s] + T.float32(1)
+
+    mod = tvm.IRModule.from_expr(before)
+    with tvm.transform.PassContext(config={"tir.enable_buffer_level_predication": False}):
+        with tvm.target.Target("llvm -mtriple=aarch64-linux-gnu -mattr=+sve"):
+            after = tvm.tir.transform.VectorizeLoop()(mod)["main"]
+    tvm.ir.assert_structural_equal(after, expected)
+

Review Comment:
   Can we also add test cases for the target override both in the function attributes (`T.func_attr({'target': T.target(...)})`) and in an attr stmt (`T.attr(T.target(...), "target", 0)`)?
   
   ([Link](https://github.com/apache/tvm/blob/main/tests/python/tir-transform/test_tir_transform_annotate_device_regions.py#L27) to example for the TVMScript syntax.)



##########
src/tir/transforms/vectorize_loop.cc:
##########
@@ -72,6 +72,126 @@ inline PrimExpr BroadcastTo(PrimExpr e, int lanes, bool is_scalable) {
   return Broadcast(e, CreateNewLanes(is_scalable, lanes));
 }
 
+bool EnableBufferLevelPredication() {

Review Comment:
   This will require updating `arith::TargetHasSVE()` as well.  Apologies for not catching it earlier in #16893.



##########
include/tvm/tir/expr.h:
##########
@@ -675,7 +679,8 @@ class BufferLoadNode : public PrimExprNode {
  */
 class BufferLoad : public PrimExpr {
  public:
-  TVM_DLL explicit BufferLoad(Buffer buffer, Array<PrimExpr> indices, Span span = Span());
+  TVM_DLL explicit BufferLoad(Buffer buffer, Array<PrimExpr> indices,
+                              PrimExpr predicate = PrimExpr(), Span span = Span());

Review Comment:
   Using `PrimExpr()` as the default would have an undefined object as the predicate.  While this is legal, it is rather unexpected.  While changing `PrimExpr` to be non-nullable would involve more breakage than I'd want to handle at the moment, we should avoid introducing additional cases where `PrimExpr` contains null.
   
   Changing this to `Optional<PrimExpr> predicate = NullOpt` would avoid this issue.



##########
include/tvm/tir/stmt.h:
##########
@@ -261,7 +265,7 @@ class BufferStoreNode : public StmtNode {
 class BufferStore : public Stmt {
  public:
   TVM_DLL explicit BufferStore(Buffer buffer, PrimExpr value, Array<PrimExpr> indices,
-                               Span span = Span());
+                               PrimExpr predicate = PrimExpr(), Span span = Span());

Review Comment:
   Same request here, `Optional<PrimExpr> predicate = NullOpt`.



##########
python/tvm/tir/expr.py:
##########
@@ -1093,20 +1093,27 @@ class BufferLoad(PrimExprWithOp):
         The buffer to be loaded.
 
     indices : List[PrimExpr]
-        The buffer indices.
+        The buffer indices to load values from.
 
     span : Optional[Span]
         The location of this expression in the source code.
+
+    predicate : Optional[PrimExpr]
+        A vector mask of int1 values indicating which lanes of a vector are to be loaded.

Review Comment:
   At first glance, the `int1` reads like a typo.  Can we explicitly call it `boolean values` instead?



##########
tests/python/codegen/test_target_codegen_aarch64.py:
##########
@@ -700,5 +700,31 @@ def before(a: T.handle):
     assert "get.active.lane.mask" in ll
 
 
+@pytest.mark.skipif(

Review Comment:
   This validates that we have the correct output for architectures that support SVE, but it doesn't test the behavior of other targets that do not (yet) support predicated loads/stores.  While the `VectorizeLoop` pass would only insert a predicated load/store for targets that support it, the predicated load/store could still be generated in hand-written kernels, or through other transforms in the future.
   
   Can we add a test, parametrized over each target tested in CI, which attempts to compile a PrimFunc containing predicated loads/stores?  For each target that supports sve, `tvm.build` should compile without error, and for each target that does not, `tvm.build` should raise an exception.
   
   



##########
src/tir/transforms/vectorize_loop.cc:
##########
@@ -72,6 +72,126 @@ inline PrimExpr BroadcastTo(PrimExpr e, int lanes, bool is_scalable) {
   return Broadcast(e, CreateNewLanes(is_scalable, lanes));
 }
 
+bool EnableBufferLevelPredication() {

Review Comment:
   I like the fallback, where it checks whether a target supports SVE in order to apply the vectorization if not specified in the pass context.  However, this function should accept a `Target` argument from the calling scope rather than using `Target::Current()`, because the `Target::Current()` may be overridden. 
   
   The override can be either at the function level (`prim_func->GetAttr<Target>(tvm::attr::kTarget)`), or for individual blocks of TIR (an `AttrStmt` with `attr_stmt->attr_key==tvm::attr::kTarget`).



##########
src/target/llvm/codegen_llvm.cc:
##########
@@ -1768,11 +1774,17 @@ llvm::Value* CodeGenLLVM::VisitExpr_(const BufferLoadNode* op) {
 
   std::vector<llvm::Value*> loads;
 
-  auto make_load = [this, &loads](TypedPointer buffer_ptr, int /* subelement_i */, int alignment,
-                                  bool is_volatile) {
+  auto make_load = [this, &loads](TypedPointer buffer_ptr, int /* subelement_i */,
+                                  llvm::Value* predicate, int alignment, bool is_volatile) {
 #if TVM_LLVM_VERSION >= 110
-    auto load = builder_->CreateAlignedLoad(buffer_ptr.type, buffer_ptr.addr,
-                                            llvm::Align(alignment), is_volatile);
+    llvm::Instruction* load = nullptr;
+    if (predicate != NULL) {
+      load = builder_->CreateMaskedLoad(buffer_ptr.type, buffer_ptr.addr, llvm::Align(alignment),
+                                        predicate);
+    } else {
+      load = builder_->CreateAlignedLoad(buffer_ptr.type, buffer_ptr.addr, llvm::Align(alignment),
+                                         is_volatile);
+    }
 #elif TVM_LLVM_VERSION >= 80
     auto load =

Review Comment:
   The PR only adds `CreateMaskedLoad` when `TVM_LLVM_VERSION >= 110`.  If somebody is using an older version of LLVM, it would silently ignore the predicate for the load/store.  We should either support it, or throw an exception.
   
   It looks like `CreateMaskedLoad` has been supported in LLVM since [this commit](https://github.com/llvm/llvm-project/commit/f1de34b84dea91b5060ee0fafbadaad5deaf199c), so I'd lean toward adding it in the other `#elif` branches.



##########
python/tvm/tir/buffer.py:
##########
@@ -141,6 +141,57 @@ def vstore(self, begin, value):
         begin = (begin,) if isinstance(begin, (int, PrimExpr)) else begin
         return _ffi_api.BufferVStore(self, begin, value)  # type: ignore
 
+    def load(self, indices, predicate=None):

Review Comment:
   These look very similar to the existing `vload` and `vstore` methods.  Should they be extended instead with a `predicate` argument instead, rather than introducing new methods?



##########
python/tvm/tir/stmt.py:
##########
@@ -224,24 +224,29 @@ class BufferStore(Stmt):
     indices : List[PrimExpr]
         The indices location to be stored.
 
+    predicate : Optional[PrimExpr]
+        A vector mask of int1 values indicating which lanes of a vector are to be stored.

Review Comment:
   Same nitpick here, using `boolean` instead of `int1`.



##########
src/tir/ir/stmt.cc:
##########
@@ -458,7 +458,8 @@ TVM_REGISTER_GLOBAL("tir.Evaluate").set_body_typed([](PrimExpr value, Span span)
 TVM_REGISTER_NODE_TYPE(EvaluateNode);
 
 // BufferStore
-BufferStore::BufferStore(Buffer buffer, PrimExpr value, Array<PrimExpr> indices, Span span) {
+BufferStore::BufferStore(Buffer buffer, PrimExpr value, Array<PrimExpr> indices, PrimExpr predicate,
+                         Span span) {
   ICHECK_EQ(buffer->shape.size(), indices.size())

Review Comment:
   We should add an assert here, that if `predicate` is defined, the number of lanes in `value` and in `predicate` are identical.



##########
src/target/llvm/codegen_llvm.cc:
##########
@@ -1768,11 +1774,17 @@ llvm::Value* CodeGenLLVM::VisitExpr_(const BufferLoadNode* op) {
 
   std::vector<llvm::Value*> loads;
 
-  auto make_load = [this, &loads](TypedPointer buffer_ptr, int /* subelement_i */, int alignment,
-                                  bool is_volatile) {
+  auto make_load = [this, &loads](TypedPointer buffer_ptr, int /* subelement_i */,
+                                  llvm::Value* predicate, int alignment, bool is_volatile) {
 #if TVM_LLVM_VERSION >= 110
-    auto load = builder_->CreateAlignedLoad(buffer_ptr.type, buffer_ptr.addr,
-                                            llvm::Align(alignment), is_volatile);
+    llvm::Instruction* load = nullptr;
+    if (predicate != NULL) {
+      load = builder_->CreateMaskedLoad(buffer_ptr.type, buffer_ptr.addr, llvm::Align(alignment),
+                                        predicate);

Review Comment:
   It looks like this ignores the `is_volatile` argument.  Can we provide `is_volatile` to `CreateMaskedLoad`?  If not, we should specify that these two arguments are incompatible and throw an exception if they are both provided.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SVE] Add support for representing and creating buffer-level predicates [tvm]

Posted by "Lunderberg (via GitHub)" <gi...@apache.org>.
Lunderberg commented on code in PR #16966:
URL: https://github.com/apache/tvm/pull/16966#discussion_r1598722953


##########
python/tvm/tir/buffer.py:
##########
@@ -141,6 +141,57 @@ def vstore(self, begin, value):
         begin = (begin,) if isinstance(begin, (int, PrimExpr)) else begin
         return _ffi_api.BufferVStore(self, begin, value)  # type: ignore
 
+    def load(self, indices, predicate=None):

Review Comment:
   Ah, I see.  I think this is an unfortunate confusion resulting from the name `begin` for the parameter in `vload` and `vstore`.  This parameter would be better named `indices`, because it is used to set the `indices` argument of the buffer load/store.
   
   While there's some additional logic to handle automatic promotion from a scalar index to a vector index when needed ([link](https://github.com/apache/tvm/blob/main/src/tir/ir/buffer.cc#L410)), a vector index can also be provided directly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org