You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2020/05/15 07:42:33 UTC

[GitHub] [incubator-tvm] Menooker opened a new pull request #5601: [TVM Core] Add bfloat16

Menooker opened a new pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601


   We add bfloat16 as a new type named "bf16" in the frontend. Completed LLVM backend for generating bf16.
   
    * Use int16 as the storage type in LLVM
    * Add legalization to enable computations on bf16
    * Add runtime frontend support (e.g. allow converting numpy's uint16 array to bf16 NDArray)
   
   # Details on legalization
   
   Since most of the HW has no native support for computation on bf16, we added a pass `BF16Legalization` to use fp32 computing bf16 data. It adds `cast_to_fp32()` before each Op involing bf16 operands, and use Ops of fp32 to compute. Finally, it adds a 'cast_to_bf16()' after each Op that is altered. e.g.
   
   `add(a,b)` => `cast16(add(cast32(a), cast32(b)))`
   
   We call this phase as "BF16Promotion". It is a sub-pass of `BF16Legalization` pass.
   
   We note that this will add redundant casting. e.g.
   
   `add(a, neg(b))` => `cast16(add(cast32(a), cast32(cast16(neg(cast32(b))))) `
   
   The pattern `cast32(cast16(some_fp32_value))` can be simplified to `some_fp32_value`.
   
   Thus, we add an optimization pass after "BF16Promotion" in `BF16Legalization` pass, which eliminates redundant casts.
   
   After `BF16Legalization` pass, there will be no bf16 related computation in the AST, except casting between fp32 and bf16, bf16 value comparasion and assignment.
   
   # Casting between fp32 and bf16
   
   We follow PyTorch's bf16 [casting] (https://github.com/pytorch/pytorch/blob/master/c10/util/BFloat16.h) implementation.
    


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#issuecomment-629113893


   @zhiics @liangfu Please help review this PR. Thanks!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r427698163



##########
File path: tests/python/unittest/test_target_codegen_llvm.py
##########
@@ -710,6 +710,52 @@ def _transform(f, *_):
         module(a_, b_, c_)
         tvm.testing.assert_allclose(c_.asnumpy(), (a_.asnumpy() * 2).astype('int32'))
 
+def np_float2np_bf16(arr):
+    ''' Convert a numpy array of float to a numpy array 
+    of bf16 in uint16'''
+    orig = arr.view('<u4')
+    bias = np.bitwise_and(np.right_shift(orig, 16), 1) + 0x7FFF
+    return np.right_shift(orig + bias, 16).astype('uint16')
+
+def np_float2tvm_bf16(arr):
+    ''' Convert a numpy array of float to a TVM array 
+    of bf16'''
+    nparr = np_float2np_bf16(arr)
+    return tvm.nd.empty(nparr.shape, 'bf16').copyfrom(nparr)
+
+def np_bf162np_float(arr):
+    ''' Convert a numpy array of bf16 (uint16) to a numpy array 
+    of float'''
+    u32 = np.left_shift(arr.astype('uint32'), 16)

Review comment:
       If my understanding is correct, we don't need to care about endianness. BF16 conversions only involves getting higher-ordered bits. And the operation to get higher-ordered bits in C++/Numpy is well-defined.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r434955653



##########
File path: python/tvm/_ffi/runtime_ctypes.py
##########
@@ -58,7 +58,8 @@ class DataType(ctypes.Structure):
         0 : 'int',
         1 : 'uint',
         2 : 'float',
-        4 : 'handle'
+        4 : 'handle',
+        65: 'bfloat'

Review comment:
       May I ask what I am supposed to do here? In the newest commit, I changed `4 : 'handle'` here to 3, and let 4 mapped to bfloat. I also changed `class TypeCode(object):` and relavent C++ class to let `NULL` mapped to 21, instead of 4. I have noticed that in `class TypeCode(object):`, we have to define both `BFLOAT` and `NULL`, so I thought we needed to resolve the conflict as they were all mapped to 4.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r429022665



##########
File path: src/target/llvm/codegen_llvm.cc
##########
@@ -555,12 +558,48 @@ void CodeGenLLVM::CreateSerialFor(llvm::Value* begin, llvm::Value* end, llvm::Va
   builder_->SetInsertPoint(for_end);
 }
 
+static llvm::Value* GetInt32VectorOrScalar(
+    llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>* builder, uint32_t v,
+    int lanes) {
+  if (lanes == 1) {
+    return builder->getInt32(v);
+  } else {
+    std::vector<llvm::Constant*> consts;
+    for (int i = 0; i < lanes; i++) {
+      consts.emplace_back(builder->getInt32(v));
+    }
+    return llvm::ConstantVector::get(consts);
+  }
+}
+
 // cast operatpr
 llvm::Value* CodeGenLLVM::CreateCast(DataType from, DataType to, llvm::Value* value) {
   llvm::Type* target = DTypeToLLVMType(to);
   if (value->getType() == target) return value;
   if (to.is_handle()) {
     return builder_->CreateBitCast(value, target);
+  } else if (to.is_float() && from.is_bfloat()) {
+    CHECK_EQ(from.bits(), 16);

Review comment:
       Just 2 small questions.
   
   Did you mean totally eliminating bf16 dtype in legalization pass? This will bring much more complexity in the BF16Legalize pass, because we need to check every TIR node to replace bf16 with int16. In contrast, current impl only changes computation TIR nodes. And in the codegen, the bf16 generation is quite simple, just adding another ‘else if’ in casting node and tvm dtype to llvm type converter




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r428113769



##########
File path: include/tvm/runtime/c_runtime_api.h
##########
@@ -114,6 +114,7 @@ typedef enum {
   kTVMNNVMLast = 20U,
   // The following section of code is used for non-reserved types.
   kTVMExtReserveEnd = 64U,
+  kTVMBFloat = 65U,

Review comment:
       Such dtype is covered by allocating a DLTensor with type_code equals kBFloat, and does not need patch to the code here(needed for parameter argument passing PackedFunc).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] liangfu commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
liangfu commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r427691578



##########
File path: tests/python/unittest/test_target_codegen_llvm.py
##########
@@ -710,6 +710,52 @@ def _transform(f, *_):
         module(a_, b_, c_)
         tvm.testing.assert_allclose(c_.asnumpy(), (a_.asnumpy() * 2).astype('int32'))
 
+def np_float2np_bf16(arr):
+    ''' Convert a numpy array of float to a numpy array 
+    of bf16 in uint16'''
+    orig = arr.view('<u4')
+    bias = np.bitwise_and(np.right_shift(orig, 16), 1) + 0x7FFF
+    return np.right_shift(orig + bias, 16).astype('uint16')
+
+def np_float2tvm_bf16(arr):
+    ''' Convert a numpy array of float to a TVM array 
+    of bf16'''
+    nparr = np_float2np_bf16(arr)
+    return tvm.nd.empty(nparr.shape, 'bf16').copyfrom(nparr)
+
+def np_bf162np_float(arr):
+    ''' Convert a numpy array of bf16 (uint16) to a numpy array 
+    of float'''
+    u32 = np.left_shift(arr.astype('uint32'), 16)

Review comment:
       Can we reused the following code snippet, which preserves endianness checks?
   
   https://github.com/apache/incubator-tvm/blob/6cbda80227fc18a859c4b01f57f75abbd7a16181/3rdparty/bfloat16/bfloat16.cc#L27-L35
   
   And it has wrapper functions below.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] ZhennanQin commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
ZhennanQin commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r428486346



##########
File path: src/target/llvm/codegen_llvm.cc
##########
@@ -555,12 +558,48 @@ void CodeGenLLVM::CreateSerialFor(llvm::Value* begin, llvm::Value* end, llvm::Va
   builder_->SetInsertPoint(for_end);
 }
 
+static llvm::Value* GetInt32VectorOrScalar(
+    llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>* builder, uint32_t v,
+    int lanes) {
+  if (lanes == 1) {
+    return builder->getInt32(v);
+  } else {
+    std::vector<llvm::Constant*> consts;
+    for (int i = 0; i < lanes; i++) {
+      consts.emplace_back(builder->getInt32(v));
+    }
+    return llvm::ConstantVector::get(consts);
+  }
+}
+
 // cast operatpr
 llvm::Value* CodeGenLLVM::CreateCast(DataType from, DataType to, llvm::Value* value) {
   llvm::Type* target = DTypeToLLVMType(to);
   if (value->getType() == target) return value;
   if (to.is_handle()) {
     return builder_->CreateBitCast(value, target);
+  } else if (to.is_float() && from.is_bfloat()) {
+    CHECK_EQ(from.bits(), 16);

Review comment:
       There're 2 kinds of legalization:
   1. TIR->TIR. TIR has full ability to describe any bfloat16 operation after this PR. This legalization is introduced just because of hardware limitation that current hardware only provide few bfloat16 operations. One day when hardware has full instructions support with bfloat16, ideally this legalization can be skipped. So this legalization is a target dependent pass. 
   2. TIR->LLVM IR. I guess this is the legalization that @tqchen mentions. Because LLVM IR doesn't natively support bfloat16 , i16 will be used to replace bfloat16. In this PR, I guess this is done within codegen_llvm, not by a particular pass. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r432401610



##########
File path: src/tir/transforms/bf16_legalize.cc
##########
@@ -0,0 +1,384 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file bf16_legalize.cc
+ * \brief legalize bf16 type by adding cast_to_fp32
+ */
+
+#include <tvm/runtime/registry.h>
+#include <tvm/tir/op.h>
+#include <tvm/tir/transform.h>
+
+#include <cmath>
+#include <tuple>
+
+#include "../../arith/ir_mutator_with_analyzer.h"
+#include "../../arith/ir_visitor_with_analyzer.h"
+
+namespace tvm {
+namespace tir {
+
+using arith::Analyzer;
+using arith::IRMutatorWithAnalyzer;
+
+class BF16PromoteRewriter : public StmtExprMutator {
+ public:
+  BF16PromoteRewriter() {}
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  std::tuple<PrimExpr, PrimExpr> DoCast(PrimExpr orig_a, PrimExpr orig_b, bool* is_bfloat16) {
+    auto a = this->VisitExpr(orig_a);
+    auto b = this->VisitExpr(orig_b);
+    *is_bfloat16 = false;
+    if (a->dtype.is_bfloat16()) {
+      CHECK(b->dtype.is_bfloat16());
+      *is_bfloat16 = true;
+    } else if (b->dtype.is_bfloat16()) {
+      CHECK(a->dtype.is_bfloat16());
+      *is_bfloat16 = true;
+    }
+
+    if (*is_bfloat16) {
+      DataType fp32ty(kDLFloat, 32, 1);
+      a = CastNode::make(fp32ty, a);
+      b = CastNode::make(fp32ty, b);
+    }
+    return std::make_tuple(a, b);
+  }
+
+  PrimExpr VisitExpr_(const AddNode* op) final;
+  PrimExpr VisitExpr_(const SubNode* op) final;
+  PrimExpr VisitExpr_(const MulNode* op) final;
+  PrimExpr VisitExpr_(const DivNode* op) final;
+  PrimExpr VisitExpr_(const MinNode* op) final;
+  PrimExpr VisitExpr_(const MaxNode* op) final;
+  PrimExpr VisitExpr_(const LTNode* op) final;
+  PrimExpr VisitExpr_(const LENode* op) final;
+  PrimExpr VisitExpr_(const GTNode* op) final;
+  PrimExpr VisitExpr_(const GENode* op) final;
+};
+
+#define DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(OP, FUNC)        \
+  PrimExpr BF16PromoteRewriter::VisitExpr_(const OP* op) {       \
+    PrimExpr a, b;                                               \
+    bool is_bfloat16;                                            \
+    std::tie(a, b) = DoCast(op->a, op->b, &is_bfloat16);         \
+    if (a.same_as(op->a) && b.same_as(op->b)) {                  \
+      return GetRef<PrimExpr>(op);                               \
+    } else {                                                     \
+      auto ret = FUNC(a, b);                                     \
+      if (!is_bfloat16)                                          \
+        return ret;                                              \
+      else                                                       \
+        return CastNode::make(DataType(kTVMBFloat, 16, 1), ret); \
+    }                                                            \
+  }
+
+#define DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(OP, FUNC) \
+  PrimExpr BF16PromoteRewriter::VisitExpr_(const OP* op) {        \
+    PrimExpr a, b;                                                \
+    bool is_bfloat16;                                             \
+    std::tie(a, b) = DoCast(op->a, op->b, &is_bfloat16);          \
+    if (a.same_as(op->a) && b.same_as(op->b)) {                   \
+      return GetRef<PrimExpr>(op);                                \
+    } else {                                                      \
+      auto ret = FUNC(a, b);                                      \
+      return ret;                                                 \
+    }                                                             \
+  }
+
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(AddNode, operator+)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(SubNode, operator-)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MulNode, operator*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(DivNode, div)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MinNode, min)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MaxNode, max)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(LTNode, operator<)   // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(LENode, operator<=)  // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(GTNode, operator>)   // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(GENode, operator>=)  // NOLINT(*)
+
+/*
+ * Eliminate verbose casting between fp32 and bf16
+ * Checks if the AST has the pattern:
+ *     castto32(castto16(some_fp32_op(...)))
+ * The verbose casting is generated by BF16Promote for multiple
+ * bf16 Ops in a row. e.g.:
+ *  X[i] + Y[i] + T[i] =>
+ *  bf16((float32(bf16((float32(X[i]) + float32(Y[i])))) + float32(T[i])))
+ * After this pass:
+ *  bf16(float32(X[i]) + float32(Y[i]) + float32(T[i]))
+ */
+class BF16CastEliminationRewriter : public StmtExprMutator {
+ public:
+  BF16CastEliminationRewriter() {}
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  PrimExpr VisitExpr_(const CastNode* op) final {
+    auto op_val = StmtExprMutator::VisitExpr(op->value);
+    if (op->dtype.is_float() && op->dtype.bits() == 32) {
+      // if is cast_to_fp32, check if op->value is cast_to_fp16
+      // and op->value->value is a float32
+      if (auto innercast = op_val.as<CastNode>()) {
+        if (innercast->dtype.is_bfloat16() && innercast->value->dtype.is_float() &&
+            innercast->value->dtype.bits() == 32) {
+          return innercast->value;
+        }
+      }
+    }
+    if (op->value.same_as(op_val)) return GetRef<PrimExpr>(op);
+    return CastNode::make(op->dtype, op_val);
+  }
+};
+
+// implementation from
+// https://github.com/pytorch/pytorch/blob/master/c10/util/BFloat16.h
+inline uint16_t round_to_nearest_even(float src) {
+  if (std::isnan(src)) {
+    return UINT16_C(0x7FC0);
+  } else {
+    union {
+      uint32_t U32;
+      float F32;
+    };
+
+    F32 = src;
+    uint32_t rounding_bias = ((U32 >> 16) & 1) + UINT32_C(0x7FFF);
+    return static_cast<uint16_t>((U32 + rounding_bias) >> 16);
+  }
+}
+
+/*
+ * Lower the bf16 type to int16
+ * Lower cast between bf16 and fp32
+ * Lower bf16 FloatImm to int16
+ */
+class BF16LowerRewriter : StmtExprMutator {
+ public:
+  BF16LowerRewriter() {}
+
+  std::unordered_map<const BufferNode*, Buffer> buffer_remap;
+  std::unordered_map<const VarNode*, Var> var_remap;
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  PrimExpr VisitExpr_(const CastNode* op) final {
+    auto op_val = StmtExprMutator::VisitExpr(op->value);
+    if (op->value->dtype.is_bfloat16()) {
+      // if is cast_from_bf16, check if is to fp32
+      CHECK(op->dtype.is_float() && op->dtype.bits() == 32);
+      auto uint32_dtype = DataType(kDLUInt, 32, op_val->dtype.lanes());
+      auto uint32_v = CastNode::make(uint32_dtype, op_val);
+      return CallNode::make(op->dtype, CallNode::reinterpret, {uint32_v << 16},
+                            CallNode::PureIntrinsic);
+
+    } else if (op->dtype.is_bfloat16()) {
+      // if is cast_to_bf16, check if op->value is fp32
+      CHECK(op->value->dtype.is_float() && op->value->dtype.bits() == 32);
+      auto uint32_dtype = DataType(kDLUInt, 32, op_val->dtype.lanes());
+      auto uint32_v =
+          CallNode::make(uint32_dtype, CallNode::reinterpret, {op_val}, CallNode::PureIntrinsic);
+      auto uint16_dtype = DataType(kDLUInt, 16, op_val->dtype.lanes());
+      // uint32_t rounding_bias = ((U32 >> 16) & 1) + UINT32_C(0x7FFF);

Review comment:
       can we keep this line? I left it here to illustrate what the following lines do.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r442879205



##########
File path: include/tvm/runtime/data_type.h
##########
@@ -72,6 +73,9 @@ class DataType {
     data_.code = static_cast<uint8_t>(code);
     data_.bits = static_cast<uint8_t>(bits);
     data_.lanes = static_cast<uint16_t>(lanes);
+    if (code == kBFloat) {
+      CHECK_EQ(bits, 16);

Review comment:
       let us leave it as it is for now, we can come back to it later




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] ZhennanQin commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
ZhennanQin commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r428480499



##########
File path: src/target/llvm/codegen_llvm.cc
##########
@@ -309,6 +309,9 @@ llvm::Type* CodeGenLLVM::DTypeToLLVMType(const DataType& dtype) const {
       default:
         LOG(FATAL) << "do not support " << dtype;
     }
+  } else if (dtype.is_bfloat()) {
+    CHECK_EQ(dtype.bits(), 16);

Review comment:
       A good question. Will we treat TensorFloat-32 as bfloat20? If so, then `bits` is useful to distinguish those.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r428424469



##########
File path: src/target/llvm/codegen_llvm.cc
##########
@@ -555,12 +558,48 @@ void CodeGenLLVM::CreateSerialFor(llvm::Value* begin, llvm::Value* end, llvm::Va
   builder_->SetInsertPoint(for_end);
 }
 
+static llvm::Value* GetInt32VectorOrScalar(
+    llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>* builder, uint32_t v,
+    int lanes) {
+  if (lanes == 1) {
+    return builder->getInt32(v);
+  } else {
+    std::vector<llvm::Constant*> consts;
+    for (int i = 0; i < lanes; i++) {
+      consts.emplace_back(builder->getInt32(v));
+    }
+    return llvm::ConstantVector::get(consts);
+  }
+}
+
 // cast operatpr
 llvm::Value* CodeGenLLVM::CreateCast(DataType from, DataType to, llvm::Value* value) {
   llvm::Type* target = DTypeToLLVMType(to);
   if (value->getType() == target) return value;
   if (to.is_handle()) {
     return builder_->CreateBitCast(value, target);
+  } else if (to.is_float() && from.is_bfloat()) {
+    CHECK_EQ(from.bits(), 16);

Review comment:
       Then we cannot tell whether it is a float32 => i16 or float32 => bfloat16 casting




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker edited a comment on pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker edited a comment on pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#issuecomment-633214226


   @tqchen I have changed the legalization pass to pure TIR->TIR. Added a pass called `BF16Lowering` to replace bf16 type with uint16 and lower the casting by shifting.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r428427929



##########
File path: include/tvm/runtime/data_type.h
##########
@@ -81,6 +82,10 @@ class DataType {
   bool is_float() const { return code() == DataType::kFloat; }
   /*! \return whether type is a float16 type. */
   bool is_float16() const { return is_float() && bits() == 16; }
+  /*! \return whether type is a bfloat type. */
+  bool is_bfloat() const { return code() == DataType::kBFloat; }

Review comment:
       ok, changed

##########
File path: include/tvm/runtime/data_type.h
##########
@@ -297,6 +302,8 @@ inline const char* TypeCode2Str(int type_code) {
       return "Object";
     case kTVMObjectRValueRefArg:
       return "ObjectRValueRefArg";
+    case kTVMBFloat:
+      return "bf";

Review comment:
       ok, changed




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r441248116



##########
File path: include/tvm/runtime/data_type.h
##########
@@ -72,6 +73,9 @@ class DataType {
     data_.code = static_cast<uint8_t>(code);
     data_.bits = static_cast<uint8_t>(bits);
     data_.lanes = static_cast<uint16_t>(lanes);
+    if (code == kBFloat) {
+      CHECK_EQ(bits, 16);

Review comment:
       I understand your concern. Any suggestions for the location where we put this check? Thanks.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#issuecomment-629506336


   cc @gussmith23 might be related to BYOD


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r441250295



##########
File path: include/tvm/runtime/data_type.h
##########
@@ -372,7 +372,7 @@ inline DLDataType String2DLDataType(std::string s) {
     t.lanes = 1;
     return t;
   } else if (s.substr(0, 6) == "bfloat") {
-    t.code = kTVMBFloat;
+    t.code = kDLBfloat;

Review comment:
       changed




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r429019312



##########
File path: include/tvm/runtime/c_runtime_api.h
##########
@@ -114,6 +114,7 @@ typedef enum {
   kTVMNNVMLast = 20U,
   // The following section of code is used for non-reserved types.
   kTVMExtReserveEnd = 64U,
+  kTVMBFloat = 65U,

Review comment:
       ok I understand. will change that




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r428730836



##########
File path: src/target/llvm/codegen_llvm.cc
##########
@@ -555,12 +558,48 @@ void CodeGenLLVM::CreateSerialFor(llvm::Value* begin, llvm::Value* end, llvm::Va
   builder_->SetInsertPoint(for_end);
 }
 
+static llvm::Value* GetInt32VectorOrScalar(
+    llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>* builder, uint32_t v,
+    int lanes) {
+  if (lanes == 1) {
+    return builder->getInt32(v);
+  } else {
+    std::vector<llvm::Constant*> consts;
+    for (int i = 0; i < lanes; i++) {
+      consts.emplace_back(builder->getInt32(v));
+    }
+    return llvm::ConstantVector::get(consts);
+  }
+}
+
 // cast operatpr
 llvm::Value* CodeGenLLVM::CreateCast(DataType from, DataType to, llvm::Value* value) {
   llvm::Type* target = DTypeToLLVMType(to);
   if (value->getType() == target) return value;
   if (to.is_handle()) {
     return builder_->CreateBitCast(value, target);
+  } else if (to.is_float() && from.is_bfloat()) {
+    CHECK_EQ(from.bits(), 16);

Review comment:
       Then we should legalize the cast as well in the TIR to introduce additional cast funtions, please also refer to https://tvm.apache.org/2020/05/20/bring-your-own-datatypes for releated implemenetation




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r441250362



##########
File path: src/tir/transforms/bf16_legalize.cc
##########
@@ -0,0 +1,387 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file bf16_legalize.cc
+ * \brief legalize bf16 type by adding cast_to_fp32
+ */
+
+#include <tvm/runtime/registry.h>
+#include <tvm/tir/op.h>
+#include <tvm/tir/transform.h>
+
+#include <cmath>
+#include <tuple>
+
+#include "../../arith/ir_mutator_with_analyzer.h"
+#include "../../arith/ir_visitor_with_analyzer.h"
+
+namespace tvm {
+namespace tir {
+
+using arith::Analyzer;
+using arith::IRMutatorWithAnalyzer;
+
+class BF16PromoteRewriter : public StmtExprMutator {
+ public:
+  BF16PromoteRewriter() {}
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  std::tuple<PrimExpr, PrimExpr> DoCast(PrimExpr orig_a, PrimExpr orig_b, bool* is_bfloat16) {
+    auto a = this->VisitExpr(orig_a);
+    auto b = this->VisitExpr(orig_b);
+    *is_bfloat16 = false;
+    if (a->dtype.is_bfloat16()) {
+      CHECK(b->dtype.is_bfloat16());
+      *is_bfloat16 = true;
+    } else if (b->dtype.is_bfloat16()) {
+      CHECK(a->dtype.is_bfloat16());
+      *is_bfloat16 = true;
+    }
+
+    if (*is_bfloat16) {
+      DataType fp32ty(kDLFloat, 32, 1);
+      a = CastNode::make(fp32ty, a);
+      b = CastNode::make(fp32ty, b);
+    }
+    return std::make_tuple(a, b);
+  }
+
+  PrimExpr VisitExpr_(const AddNode* op) final;
+  PrimExpr VisitExpr_(const SubNode* op) final;
+  PrimExpr VisitExpr_(const MulNode* op) final;
+  PrimExpr VisitExpr_(const DivNode* op) final;
+  PrimExpr VisitExpr_(const MinNode* op) final;
+  PrimExpr VisitExpr_(const MaxNode* op) final;
+  PrimExpr VisitExpr_(const LTNode* op) final;
+  PrimExpr VisitExpr_(const LENode* op) final;
+  PrimExpr VisitExpr_(const GTNode* op) final;
+  PrimExpr VisitExpr_(const GENode* op) final;
+};
+
+#define DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(OP, FUNC)       \
+  PrimExpr BF16PromoteRewriter::VisitExpr_(const OP* op) {      \
+    PrimExpr a, b;                                              \
+    bool is_bfloat16;                                           \
+    std::tie(a, b) = DoCast(op->a, op->b, &is_bfloat16);        \
+    if (a.same_as(op->a) && b.same_as(op->b)) {                 \
+      return GetRef<PrimExpr>(op);                              \
+    } else {                                                    \
+      auto ret = FUNC(a, b);                                    \
+      if (!is_bfloat16)                                         \
+        return ret;                                             \
+      else                                                      \
+        return CastNode::make(DataType(kDLBfloat, 16, 1), ret); \
+    }                                                           \
+  }
+
+#define DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(OP, FUNC) \
+  PrimExpr BF16PromoteRewriter::VisitExpr_(const OP* op) {        \
+    PrimExpr a, b;                                                \
+    bool is_bfloat16;                                             \
+    std::tie(a, b) = DoCast(op->a, op->b, &is_bfloat16);          \
+    if (a.same_as(op->a) && b.same_as(op->b)) {                   \
+      return GetRef<PrimExpr>(op);                                \
+    } else {                                                      \
+      auto ret = FUNC(a, b);                                      \
+      return ret;                                                 \
+    }                                                             \
+  }
+
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(AddNode, operator+)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(SubNode, operator-)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MulNode, operator*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(DivNode, div)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MinNode, min)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MaxNode, max)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(LTNode, operator<)   // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(LENode, operator<=)  // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(GTNode, operator>)   // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(GENode, operator>=)  // NOLINT(*)
+
+/*
+ * Eliminate verbose casting between fp32 and bf16
+ * Checks if the AST has the pattern:
+ *     castto32(castto16(some_fp32_op(...)))
+ * The verbose casting is generated by BF16Promote for multiple
+ * bf16 Ops in a row. e.g.:
+ *  X[i] + Y[i] + T[i] =>
+ *  bf16((float32(bf16((float32(X[i]) + float32(Y[i])))) + float32(T[i])))
+ * After this pass:
+ *  bf16(float32(X[i]) + float32(Y[i]) + float32(T[i]))
+ */
+class BF16CastEliminationRewriter : public StmtExprMutator {
+ public:
+  BF16CastEliminationRewriter() {}
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  PrimExpr VisitExpr_(const CastNode* op) final {
+    auto op_val = StmtExprMutator::VisitExpr(op->value);
+    if (op->dtype.is_float() && op->dtype.bits() == 32) {
+      // if is cast_to_fp32, check if op->value is cast_to_fp16
+      // and op->value->value is a float32
+      if (auto innercast = op_val.as<CastNode>()) {
+        if (innercast->dtype.is_bfloat16() && innercast->value->dtype.is_float() &&
+            innercast->value->dtype.bits() == 32) {
+          return innercast->value;
+        }
+      }
+    }
+    if (op->value.same_as(op_val)) return GetRef<PrimExpr>(op);
+    return CastNode::make(op->dtype, op_val);
+  }
+};
+
+// implementation from
+// https://github.com/pytorch/pytorch/blob/master/c10/util/BFloat16.h
+inline uint16_t round_to_nearest_even(float src) {

Review comment:
       changed




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r428730836



##########
File path: src/target/llvm/codegen_llvm.cc
##########
@@ -555,12 +558,48 @@ void CodeGenLLVM::CreateSerialFor(llvm::Value* begin, llvm::Value* end, llvm::Va
   builder_->SetInsertPoint(for_end);
 }
 
+static llvm::Value* GetInt32VectorOrScalar(
+    llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>* builder, uint32_t v,
+    int lanes) {
+  if (lanes == 1) {
+    return builder->getInt32(v);
+  } else {
+    std::vector<llvm::Constant*> consts;
+    for (int i = 0; i < lanes; i++) {
+      consts.emplace_back(builder->getInt32(v));
+    }
+    return llvm::ConstantVector::get(consts);
+  }
+}
+
 // cast operatpr
 llvm::Value* CodeGenLLVM::CreateCast(DataType from, DataType to, llvm::Value* value) {
   llvm::Type* target = DTypeToLLVMType(to);
   if (value->getType() == target) return value;
   if (to.is_handle()) {
     return builder_->CreateBitCast(value, target);
+  } else if (to.is_float() && from.is_bfloat()) {
+    CHECK_EQ(from.bits(), 16);

Review comment:
       Then we should legalize the cast as well in the TIR to introduce the actual impl of the cast funtions in TIR, please also refer to https://tvm.apache.org/2020/05/20/bring-your-own-datatypes for releated implemenetation




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r432964330



##########
File path: include/tvm/runtime/c_runtime_api.h
##########
@@ -114,6 +114,7 @@ typedef enum {
   kTVMNNVMLast = 20U,
   // The following section of code is used for non-reserved types.
   kTVMExtReserveEnd = 64U,
+  kTVMBFloat = 65U,

Review comment:
       plase remove the flag




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r428729913



##########
File path: include/tvm/runtime/c_runtime_api.h
##########
@@ -114,6 +114,7 @@ typedef enum {
   kTVMNNVMLast = 20U,
   // The following section of code is used for non-reserved types.
   kTVMExtReserveEnd = 64U,
+  kTVMBFloat = 65U,

Review comment:
       I don't think you will `kTVMBFloat` to support this feature. The kDLBFloat flag in the runtime::DataType should be sufficient for NDArray contents(because the type code in the NDArray contents diverges from the TVM type code above the OpaqueHandle).

##########
File path: include/tvm/runtime/c_runtime_api.h
##########
@@ -114,6 +114,7 @@ typedef enum {
   kTVMNNVMLast = 20U,
   // The following section of code is used for non-reserved types.
   kTVMExtReserveEnd = 64U,
+  kTVMBFloat = 65U,

Review comment:
       I don't think you will `kTVMBFloat` to support this feature. The kDLBFloat flag in the runtime::DataType should be sufficient for NDArray contents(because the runtime::DataType's type code in the NDArray contents diverges from the TVM type code above the OpaqueHandle).

##########
File path: include/tvm/runtime/c_runtime_api.h
##########
@@ -114,6 +114,7 @@ typedef enum {
   kTVMNNVMLast = 20U,
   // The following section of code is used for non-reserved types.
   kTVMExtReserveEnd = 64U,
+  kTVMBFloat = 65U,

Review comment:
       I don't think you will `kTVMBFloat` to support this feature. The DataType::kDLBFloat flag in the runtime::DataType should be sufficient for NDArray contents(because the runtime::DataType's type code in the NDArray contents diverges from the TVM type code above the OpaqueHandle).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] junrushao1994 commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
junrushao1994 commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r439716507



##########
File path: tests/python/unittest/test_tir_transform_bf16_legalize.py
##########
@@ -0,0 +1,152 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import tvm
+import topi
+from tvm import te
+from tvm.tir import const
+
+
+def lower_stmt(sche, params, passfunc):
+    func = tvm.driver.build_module.form_irmodule(sche, params, "main", None)["main"]
+    func = passfunc()(
+        tvm.IRModule.from_expr(func))["main"]
+    stmt = func.body
+    return stmt
+
+
+def test_promote():
+    def runpass(op, passfunc):
+        a = te.placeholder((100,), dtype='bfloat16')
+        b = te.placeholder((100,), dtype='bfloat16')
+        c = te.compute((100,), lambda i: op(a[i], b[i]))
+        s = te.create_schedule(c.op)
+        return lower_stmt(s, [a, b, c], passfunc)
+    
+    def get_promoted(op):
+        a = te.placeholder((100,), dtype='bfloat16')
+        b = te.placeholder((100,), dtype='bfloat16')
+        c = te.compute((100,), lambda i:
+                topi.cast(op(topi.cast(a[i],'float'),
+                    topi.cast(b[i],'float')), 'bfloat16')
+                )
+        s = te.create_schedule(c.op)
+        func = tvm.driver.build_module.form_irmodule(s, [a,b,c], "main", None)["main"]
+        return func.body
+
+    def test_promoted(op):
+        stmt = runpass(op, tvm.tir.transform.BF16Promote)
+        tvm.ir.assert_structural_equal(stmt, get_promoted(op))
+    test_promoted(topi.add)
+    test_promoted(topi.subtract)
+    test_promoted(topi.multiply)
+    test_promoted(topi.divide)
+
+def test_eliminate():
+    def to32(v):
+        return topi.cast(v, 'float')
+    def to16(v):
+        return topi.cast(v, 'bfloat16')
+    def get_eliminated():
+        a = te.placeholder((100,), dtype='bfloat16')
+        b = te.placeholder((100,), dtype='bfloat16')
+        c = te.compute((100,), lambda i: to16(
+            topi.add(
+                to32(
+                    to16(
+                        topi.add(
+                            to32(a[i]),
+                            to32(b[i]),
+                        )
+                    )
+                ),
+                to32(
+                    to16(
+                        topi.add(
+                            to32(a[i]),
+                            to32(b[i]),
+                        )
+                    )
+                )
+            )
+        ))
+        s = te.create_schedule(c.op)
+        stmt = lower_stmt(s, [a, b, c], tvm.tir.transform.BF16CastElimination)
+        return stmt
+
+    def get_target():
+        a = te.placeholder((100,), dtype='bfloat16')
+        b = te.placeholder((100,), dtype='bfloat16')
+        c = te.compute((100,), lambda i: to16(
+            topi.add(topi.add(
+                        to32(a[i]),
+                        to32(b[i]),
+                    ),
+                    topi.add(
+                        to32(a[i]),
+                        to32(b[i]),
+                    )
+                )
+        ))
+        s = te.create_schedule(c.op)
+        func = tvm.driver.build_module.form_irmodule(s, [a,b,c], "main", None)["main"]
+        return func.body
+    tvm.ir.assert_structural_equal(get_eliminated(), get_target())
+
+def test_legalize():
+    def to32(v):
+        uint32_v = topi.cast(v, "uint32")
+        uint32_v = tvm.tir.call_pure_intrin("uint32", "shift_left", uint32_v, tvm.tir.const(16, "uint32"))
+        return tvm.tir.call_pure_intrin("float32", "reinterpret", uint32_v)
+    def to16(v):
+        uint32_v = tvm.tir.call_pure_intrin("uint32", "reinterpret", v)
+        rounding_bias = tvm.tir.call_pure_intrin("uint32", "shift_right", uint32_v, tvm.tir.const(16, "uint32"))
+        rounding_bias = tvm.tir.call_pure_intrin("uint32", "bitwise_and", rounding_bias, tvm.tir.const(1, "uint32"))
+        rounding_bias = rounding_bias + tvm.tir.const(0x7FFF, "uint16")
+        uint32_v = uint32_v + rounding_bias
+        uint32_v = tvm.tir.call_pure_intrin("uint32", "shift_right", uint32_v, tvm.tir.const(16, "uint32"))
+        return topi.cast(uint32_v, 'uint16')
+
+    def check(fcompute_before, fcompute_after):
+        a = te.placeholder((100,), dtype='bfloat16', name = 'A')
+        b = te.placeholder((100,), dtype='bfloat16', name = 'B')
+        c = te.compute((100,), fcompute_before(a,b), name = 'C')
+        s = te.create_schedule(c.op)
+        stmt = lower_stmt(s, [a, b, c], tvm.tir.transform.BF16Legalize)
+
+        a = te.placeholder((100,), dtype='uint16', name = 'A')
+        b = te.placeholder((100,), dtype='uint16', name = 'B')
+        c = te.compute((100,), fcompute_after(a,b), name = 'C')
+        s = te.create_schedule(c.op)
+        func = tvm.driver.build_module.form_irmodule(s, [a,b,c], "main", None)["main"]
+        tvm.ir.assert_structural_equal(stmt, func.body)
+
+    def orig1(a,b):
+        return lambda i: a[i]+b[i]+a[99-i]+b[99-i]
+    def after1(a,b):
+        return lambda i: to16(to32(a[i])+to32(b[i])+to32(a[99-i])+to32(b[99-i]))
+    def orig2(a,b):
+        return lambda i: a[i]*b[i]+a[99-i]*b[99-i]+a[i]
+    def after2(a,b):
+        return lambda i: to16(to32(a[i])*to32(b[i])+to32(a[99-i])*to32(b[99-i])+to32(a[i]))

Review comment:
       Ooops I see. That makes sense then :-)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r432964447



##########
File path: src/tir/transforms/bf16_legalize.cc
##########
@@ -0,0 +1,387 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file bf16_legalize.cc
+ * \brief legalize bf16 type by adding cast_to_fp32
+ */
+
+#include <tvm/runtime/registry.h>
+#include <tvm/tir/op.h>
+#include <tvm/tir/transform.h>
+
+#include <cmath>
+#include <tuple>
+
+#include "../../arith/ir_mutator_with_analyzer.h"
+#include "../../arith/ir_visitor_with_analyzer.h"
+
+namespace tvm {
+namespace tir {
+
+using arith::Analyzer;
+using arith::IRMutatorWithAnalyzer;
+
+class BF16PromoteRewriter : public StmtExprMutator {
+ public:
+  BF16PromoteRewriter() {}
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  std::tuple<PrimExpr, PrimExpr> DoCast(PrimExpr orig_a, PrimExpr orig_b, bool* is_bfloat16) {
+    auto a = this->VisitExpr(orig_a);
+    auto b = this->VisitExpr(orig_b);
+    *is_bfloat16 = false;
+    if (a->dtype.is_bfloat16()) {
+      CHECK(b->dtype.is_bfloat16());
+      *is_bfloat16 = true;
+    } else if (b->dtype.is_bfloat16()) {
+      CHECK(a->dtype.is_bfloat16());
+      *is_bfloat16 = true;
+    }
+
+    if (*is_bfloat16) {
+      DataType fp32ty(kDLFloat, 32, 1);
+      a = CastNode::make(fp32ty, a);
+      b = CastNode::make(fp32ty, b);
+    }
+    return std::make_tuple(a, b);
+  }
+
+  PrimExpr VisitExpr_(const AddNode* op) final;
+  PrimExpr VisitExpr_(const SubNode* op) final;
+  PrimExpr VisitExpr_(const MulNode* op) final;
+  PrimExpr VisitExpr_(const DivNode* op) final;
+  PrimExpr VisitExpr_(const MinNode* op) final;
+  PrimExpr VisitExpr_(const MaxNode* op) final;
+  PrimExpr VisitExpr_(const LTNode* op) final;
+  PrimExpr VisitExpr_(const LENode* op) final;
+  PrimExpr VisitExpr_(const GTNode* op) final;
+  PrimExpr VisitExpr_(const GENode* op) final;
+};
+
+#define DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(OP, FUNC)        \
+  PrimExpr BF16PromoteRewriter::VisitExpr_(const OP* op) {       \
+    PrimExpr a, b;                                               \
+    bool is_bfloat16;                                            \
+    std::tie(a, b) = DoCast(op->a, op->b, &is_bfloat16);         \
+    if (a.same_as(op->a) && b.same_as(op->b)) {                  \
+      return GetRef<PrimExpr>(op);                               \
+    } else {                                                     \
+      auto ret = FUNC(a, b);                                     \
+      if (!is_bfloat16)                                          \
+        return ret;                                              \
+      else                                                       \
+        return CastNode::make(DataType(kTVMBFloat, 16, 1), ret); \
+    }                                                            \
+  }
+
+#define DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(OP, FUNC) \
+  PrimExpr BF16PromoteRewriter::VisitExpr_(const OP* op) {        \
+    PrimExpr a, b;                                                \
+    bool is_bfloat16;                                             \
+    std::tie(a, b) = DoCast(op->a, op->b, &is_bfloat16);          \
+    if (a.same_as(op->a) && b.same_as(op->b)) {                   \
+      return GetRef<PrimExpr>(op);                                \
+    } else {                                                      \
+      auto ret = FUNC(a, b);                                      \
+      return ret;                                                 \
+    }                                                             \
+  }
+
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(AddNode, operator+)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(SubNode, operator-)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MulNode, operator*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(DivNode, div)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MinNode, min)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MaxNode, max)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(LTNode, operator<)   // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(LENode, operator<=)  // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(GTNode, operator>)   // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(GENode, operator>=)  // NOLINT(*)
+
+/*
+ * Eliminate verbose casting between fp32 and bf16
+ * Checks if the AST has the pattern:
+ *     castto32(castto16(some_fp32_op(...)))
+ * The verbose casting is generated by BF16Promote for multiple
+ * bf16 Ops in a row. e.g.:
+ *  X[i] + Y[i] + T[i] =>
+ *  bf16((float32(bf16((float32(X[i]) + float32(Y[i])))) + float32(T[i])))
+ * After this pass:
+ *  bf16(float32(X[i]) + float32(Y[i]) + float32(T[i]))
+ */
+class BF16CastEliminationRewriter : public StmtExprMutator {
+ public:
+  BF16CastEliminationRewriter() {}
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  PrimExpr VisitExpr_(const CastNode* op) final {
+    auto op_val = StmtExprMutator::VisitExpr(op->value);
+    if (op->dtype.is_float() && op->dtype.bits() == 32) {
+      // if is cast_to_fp32, check if op->value is cast_to_fp16
+      // and op->value->value is a float32
+      if (auto innercast = op_val.as<CastNode>()) {
+        if (innercast->dtype.is_bfloat16() && innercast->value->dtype.is_float() &&
+            innercast->value->dtype.bits() == 32) {
+          return innercast->value;
+        }
+      }
+    }
+    if (op->value.same_as(op_val)) return GetRef<PrimExpr>(op);
+    return CastNode::make(op->dtype, op_val);
+  }
+};
+
+// implementation from
+// https://github.com/pytorch/pytorch/blob/master/c10/util/BFloat16.h
+inline uint16_t round_to_nearest_even(float src) {
+  if (std::isnan(src)) {
+    return UINT16_C(0x7FC0);
+  } else {
+    union {
+      uint32_t U32;
+      float F32;
+    };
+
+    F32 = src;
+    uint32_t rounding_bias = ((U32 >> 16) & 1) + UINT32_C(0x7FFF);
+    return static_cast<uint16_t>((U32 + rounding_bias) >> 16);
+  }
+}
+
+/*
+ * Lower the bf16 type to int16
+ * Lower cast between bf16 and fp32
+ * Lower bf16 FloatImm to int16
+ */
+class BF16LowerRewriter : StmtExprMutator {
+ public:
+  BF16LowerRewriter() {}
+
+  std::unordered_map<const BufferNode*, Buffer> buffer_remap;
+  std::unordered_map<const VarNode*, Var> var_remap;
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  PrimExpr VisitExpr_(const CastNode* op) final {
+    auto op_val = StmtExprMutator::VisitExpr(op->value);
+    if (op->value->dtype.is_bfloat16()) {
+      // if is cast_from_bf16, check if is to fp32
+      CHECK(op->dtype.is_float() && op->dtype.bits() == 32);
+      auto uint32_dtype = DataType(kDLUInt, 32, op_val->dtype.lanes());
+      auto uint32_v = CastNode::make(uint32_dtype, op_val);
+      // to be endian invariant.
+      return CallNode::make(op->dtype, CallNode::reinterpret, {uint32_v << 16},
+                            CallNode::PureIntrinsic);
+
+    } else if (op->dtype.is_bfloat16()) {
+      // if is cast_to_bf16, check if op->value is fp32
+      CHECK(op->value->dtype.is_float() && op->value->dtype.bits() == 32);
+      auto uint32_dtype = DataType(kDLUInt, 32, op_val->dtype.lanes());
+      auto uint32_v =
+          CallNode::make(uint32_dtype, CallNode::reinterpret, {op_val}, CallNode::PureIntrinsic);
+      auto uint16_dtype = DataType(kDLUInt, 16, op_val->dtype.lanes());
+      /* the following TIR is equivalent to the C++ code below:
+      uint32_t rounding_bias = ((U32 >> 16) & 1) + UINT32_C(0x7FFF);
+      return static_cast<uint16_t>((U32 + rounding_bias) >> 16);*/
+      auto rounding_bias = ((uint32_v >> 16) & 1) + make_const(uint16_dtype, 0x7FFF);
+      // to be endian invariant.
+      return CastNode::make(uint16_dtype, {(uint32_v + rounding_bias) >> 16});
+    }
+    if (op->value.same_as(op_val)) return GetRef<PrimExpr>(op);
+    return CastNode::make(op->dtype, op_val);
+  }
+
+  PrimExpr VisitExpr_(const VarNode* op) final {
+    auto itr = var_remap.find(op);
+    if (itr != var_remap.end()) {
+      return itr->second;
+    }
+    if (op->dtype.is_bfloat16()) {
+      CHECK(!op->type_annotation.defined());
+      auto ret = Var(op->name_hint, op->dtype);
+      var_remap[op] = ret;
+      return std::move(ret);
+    }
+    return StmtExprMutator::VisitExpr_(op);
+  }
+
+  Stmt VisitStmt_(const AllocateNode* op) final {
+    Stmt node_holder;
+    const AllocateNode* newop;
+    if (op->dtype.is_bfloat16()) {
+      auto v = AllocateNode::make(op->buffer_var, DataType::UInt(16, op->dtype.lanes()),
+                                  op->extents, op->condition, op->body);
+      node_holder = v;
+      newop = static_cast<const AllocateNode*>(v.operator->());
+    } else {
+      newop = op;
+    }
+    return StmtExprMutator::VisitStmt_(newop);
+  }
+
+  Stmt VisitStmt_(const BufferStoreNode* op) final {
+    auto itr = buffer_remap.find(op->buffer.operator->());
+    const BufferStoreNode* newop;
+    BufferStore newop_holder;
+    if (itr != buffer_remap.end()) {
+      newop_holder = BufferStore(itr->second, op->value, op->indices);
+      newop = newop_holder.operator->();
+    } else {
+      newop = op;
+    }
+    return StmtExprMutator::VisitStmt_(newop);
+  }
+
+  Stmt VisitStmt_(const AttrStmtNode* op) final {
+    const AttrStmtNode* newop = op;
+    Stmt newop_holder;
+    if (auto buffer = op->node.as<BufferNode>()) {
+      auto itr = buffer_remap.find(buffer);
+      if (itr != buffer_remap.end()) {
+        newop_holder = AttrStmtNode::make(itr->second, op->attr_key, op->value, op->body);
+        newop = newop_holder.as<AttrStmtNode>();
+      }
+    } else if (auto buffer = op->node.as<VarNode>()) {
+      auto itr = var_remap.find(buffer);
+      if (itr != var_remap.end()) {
+        newop_holder = AttrStmtNode::make(itr->second, op->attr_key, op->value, op->body);
+        newop = newop_holder.as<AttrStmtNode>();
+      }
+    }
+    return StmtExprMutator::VisitStmt_(newop);
+  }
+
+  Stmt VisitStmt_(const BufferRealizeNode* op) final {
+    auto itr = buffer_remap.find(op->buffer.operator->());
+    const BufferRealizeNode* newop;
+    Stmt newop_holder;
+    if (itr != buffer_remap.end()) {
+      auto v = BufferRealize(itr->second, op->bounds, op->condition, op->body);
+      newop_holder = v;
+      newop = v.operator->();
+    } else {
+      newop = op;
+    }
+    return StmtExprMutator::VisitStmt_(newop);
+  }
+
+  PrimExpr VisitExpr_(const BufferLoadNode* op) final {
+    auto itr = buffer_remap.find(op->buffer.operator->());
+    const BufferLoadNode* newop;
+    BufferLoad newop_holder;
+    if (itr != buffer_remap.end()) {
+      newop_holder = BufferLoad(itr->second, op->indices);
+      newop = newop_holder.operator->();
+    } else {
+      newop = op;
+    }
+    return StmtExprMutator::VisitExpr_(newop);
+  }
+
+  PrimExpr VisitExpr_(const LoadNode* op) final {
+    bool is_bf16 = false;
+    if (op->dtype.is_bfloat16()) {
+      is_bf16 = true;
+    }
+    PrimExpr index = this->VisitExpr(op->index);
+    PrimExpr predicate = this->VisitExpr(op->predicate);
+    if (index.same_as(op->index) && predicate.same_as(op->predicate) && !is_bf16) {
+      return GetRef<PrimExpr>(op);
+    } else {
+      return LoadNode::make(is_bf16 ? DataType::UInt(16, op->dtype.lanes()) : op->dtype,
+                            op->buffer_var, index, predicate);
+    }
+  }
+
+  PrimExpr VisitExpr_(const FloatImmNode* op) final {
+    if (op->dtype.is_bfloat16()) {
+      return IntImm(DataType::UInt(16, op->dtype.lanes()),
+                    round_to_nearest_even(static_cast<float>(op->value)));
+    }
+    return StmtExprMutator::VisitExpr_(op);
+  }
+
+  void alter_buffers(PrimFuncNode* op) {

Review comment:
       Code style, use CamelCase as per Google C Style

##########
File path: python/tvm/_ffi/runtime_ctypes.py
##########
@@ -58,7 +58,8 @@ class DataType(ctypes.Structure):
         0 : 'int',
         1 : 'uint',
         2 : 'float',
-        4 : 'handle'
+        4 : 'handle',
+        65: 'bfloat'

Review comment:
       use a code that is consistent with dlpack

##########
File path: include/tvm/runtime/data_type.h
##########
@@ -46,6 +46,7 @@ class DataType {
     kUInt = kDLUInt,
     kFloat = kDLFloat,
     kHandle = TVMTypeCode::kTVMOpaqueHandle,
+    kBFloat = kTVMBFloat,

Review comment:
       Use kDLBfloat as it has already been accepted to the latest dlpack




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#issuecomment-635725119


   @liangfu please followup with the reviews. @gussmith23 it would be great if you can also help to take a look


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r427756975



##########
File path: src/target/llvm/codegen_llvm.cc
##########
@@ -555,12 +558,48 @@ void CodeGenLLVM::CreateSerialFor(llvm::Value* begin, llvm::Value* end, llvm::Va
   builder_->SetInsertPoint(for_end);
 }
 
+static llvm::Value* GetInt32VectorOrScalar(
+    llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>* builder, uint32_t v,
+    int lanes) {
+  if (lanes == 1) {
+    return builder->getInt32(v);
+  } else {
+    std::vector<llvm::Constant*> consts;
+    for (int i = 0; i < lanes; i++) {
+      consts.emplace_back(builder->getInt32(v));
+    }
+    return llvm::ConstantVector::get(consts);
+  }
+}
+
 // cast operatpr
 llvm::Value* CodeGenLLVM::CreateCast(DataType from, DataType to, llvm::Value* value) {
   llvm::Type* target = DTypeToLLVMType(to);
   if (value->getType() == target) return value;
   if (to.is_handle()) {
     return builder_->CreateBitCast(value, target);
+  } else if (to.is_float() && from.is_bfloat()) {
+    CHECK_EQ(from.bits(), 16);

Review comment:
       We are actually doing TIR=>TIR legalization pass in TVM. See src/tir/transforms/bf16_legalize.cc 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#issuecomment-636580788


   Some changes to the type code: We now use kDLBfloat defined in DLPack, which equals to 4. It has a conflict with old type code `kTVMNullptr = 4U`, so I move it to `kTVMNullptr = 21U'


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r427307195



##########
File path: tests/python/unittest/test_target_codegen_llvm.py
##########
@@ -710,6 +710,52 @@ def _transform(f, *_):
         module(a_, b_, c_)
         tvm.testing.assert_allclose(c_.asnumpy(), (a_.asnumpy() * 2).astype('int32'))
 
+def np_float2np_bf16(arr):
+    ''' Convert a numpy array of float to a numpy array 
+    of bf16 in uint16'''
+    orig = arr.view('<u4')
+    bias = np.bitwise_and(np.right_shift(orig, 16), 1) + 0x7FFF
+    return np.right_shift(orig + bias, 16).astype('uint16')
+
+def np_float2tvm_bf16(arr):
+    ''' Convert a numpy array of float to a TVM array 
+    of bf16'''
+    nparr = np_float2np_bf16(arr)
+    return tvm.nd.empty(nparr.shape, 'bf16').copyfrom(nparr)
+
+def np_bf162np_float(arr):
+    ''' Convert a numpy array of bf16 (uint16) to a numpy array 
+    of float'''
+    u32 = np.left_shift(arr.astype('uint32'), 16)

Review comment:
       I am not 100% sure about this. I have tested the code on x86, not (yet) on other arch.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r438430324



##########
File path: src/tir/transforms/bf16_legalize.cc
##########
@@ -0,0 +1,387 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file bf16_legalize.cc
+ * \brief legalize bf16 type by adding cast_to_fp32
+ */
+
+#include <tvm/runtime/registry.h>
+#include <tvm/tir/op.h>
+#include <tvm/tir/transform.h>
+
+#include <cmath>
+#include <tuple>
+
+#include "../../arith/ir_mutator_with_analyzer.h"
+#include "../../arith/ir_visitor_with_analyzer.h"
+
+namespace tvm {
+namespace tir {
+
+using arith::Analyzer;
+using arith::IRMutatorWithAnalyzer;
+
+class BF16PromoteRewriter : public StmtExprMutator {
+ public:
+  BF16PromoteRewriter() {}
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  std::tuple<PrimExpr, PrimExpr> DoCast(PrimExpr orig_a, PrimExpr orig_b, bool* is_bfloat16) {
+    auto a = this->VisitExpr(orig_a);
+    auto b = this->VisitExpr(orig_b);
+    *is_bfloat16 = false;
+    if (a->dtype.is_bfloat16()) {
+      CHECK(b->dtype.is_bfloat16());
+      *is_bfloat16 = true;
+    } else if (b->dtype.is_bfloat16()) {
+      CHECK(a->dtype.is_bfloat16());
+      *is_bfloat16 = true;
+    }
+
+    if (*is_bfloat16) {
+      DataType fp32ty(kDLFloat, 32, 1);
+      a = CastNode::make(fp32ty, a);
+      b = CastNode::make(fp32ty, b);
+    }
+    return std::make_tuple(a, b);
+  }
+
+  PrimExpr VisitExpr_(const AddNode* op) final;
+  PrimExpr VisitExpr_(const SubNode* op) final;
+  PrimExpr VisitExpr_(const MulNode* op) final;
+  PrimExpr VisitExpr_(const DivNode* op) final;
+  PrimExpr VisitExpr_(const MinNode* op) final;
+  PrimExpr VisitExpr_(const MaxNode* op) final;
+  PrimExpr VisitExpr_(const LTNode* op) final;
+  PrimExpr VisitExpr_(const LENode* op) final;
+  PrimExpr VisitExpr_(const GTNode* op) final;
+  PrimExpr VisitExpr_(const GENode* op) final;
+};
+
+#define DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(OP, FUNC)       \
+  PrimExpr BF16PromoteRewriter::VisitExpr_(const OP* op) {      \
+    PrimExpr a, b;                                              \
+    bool is_bfloat16;                                           \
+    std::tie(a, b) = DoCast(op->a, op->b, &is_bfloat16);        \
+    if (a.same_as(op->a) && b.same_as(op->b)) {                 \
+      return GetRef<PrimExpr>(op);                              \
+    } else {                                                    \
+      auto ret = FUNC(a, b);                                    \
+      if (!is_bfloat16)                                         \
+        return ret;                                             \
+      else                                                      \
+        return CastNode::make(DataType(kDLBfloat, 16, 1), ret); \
+    }                                                           \
+  }
+
+#define DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(OP, FUNC) \
+  PrimExpr BF16PromoteRewriter::VisitExpr_(const OP* op) {        \
+    PrimExpr a, b;                                                \
+    bool is_bfloat16;                                             \
+    std::tie(a, b) = DoCast(op->a, op->b, &is_bfloat16);          \
+    if (a.same_as(op->a) && b.same_as(op->b)) {                   \
+      return GetRef<PrimExpr>(op);                                \
+    } else {                                                      \
+      auto ret = FUNC(a, b);                                      \
+      return ret;                                                 \
+    }                                                             \
+  }
+
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(AddNode, operator+)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(SubNode, operator-)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MulNode, operator*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(DivNode, div)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MinNode, min)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MaxNode, max)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(LTNode, operator<)   // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(LENode, operator<=)  // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(GTNode, operator>)   // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(GENode, operator>=)  // NOLINT(*)
+
+/*
+ * Eliminate verbose casting between fp32 and bf16
+ * Checks if the AST has the pattern:
+ *     castto32(castto16(some_fp32_op(...)))
+ * The verbose casting is generated by BF16Promote for multiple
+ * bf16 Ops in a row. e.g.:
+ *  X[i] + Y[i] + T[i] =>
+ *  bf16((float32(bf16((float32(X[i]) + float32(Y[i])))) + float32(T[i])))
+ * After this pass:
+ *  bf16(float32(X[i]) + float32(Y[i]) + float32(T[i]))
+ */
+class BF16CastEliminationRewriter : public StmtExprMutator {
+ public:
+  BF16CastEliminationRewriter() {}
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  PrimExpr VisitExpr_(const CastNode* op) final {
+    auto op_val = StmtExprMutator::VisitExpr(op->value);
+    if (op->dtype.is_float() && op->dtype.bits() == 32) {
+      // if is cast_to_fp32, check if op->value is cast_to_fp16
+      // and op->value->value is a float32
+      if (auto innercast = op_val.as<CastNode>()) {
+        if (innercast->dtype.is_bfloat16() && innercast->value->dtype.is_float() &&
+            innercast->value->dtype.bits() == 32) {
+          return innercast->value;
+        }
+      }
+    }
+    if (op->value.same_as(op_val)) return GetRef<PrimExpr>(op);
+    return CastNode::make(op->dtype, op_val);
+  }
+};
+
+// implementation from
+// https://github.com/pytorch/pytorch/blob/master/c10/util/BFloat16.h
+inline uint16_t round_to_nearest_even(float src) {

Review comment:
       Google C Style, we cannot directly copy code from another codebase into the mainline, we would need to either put it in 3rdparty, or implement it independently




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#issuecomment-632164851


   Given that this is a new feature that will affect quite some people, please open a new RFC thread in the discuss forum to describe the motivation and the high level design. Thank you!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#issuecomment-633214226


   @tqchen I have changed the legalization pass to pure TIR->TIR. Added a pass called `BF16Lowering` to replace bf16 type with uint16 and lower the casting by shifting.
   
   I have met a problem that I need to replace `BufferNode` of bf16 with uint16. It works well and can produce same result as numpy. However, a lowered bf16 cast cannot pass `assert_structural_equal` test, like [here](https://github.com/Menooker/incubator-tvm/blob/177f99a8ce7f95e927cabeaf18684ade0c8ebc73/tests/python/unittest/test_tir_transform_bf16_legalize.py#L143). Is this a problem of this PR?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r428427975



##########
File path: include/tvm/runtime/data_type.h
##########
@@ -297,6 +302,8 @@ inline const char* TypeCode2Str(int type_code) {
       return "Object";
     case kTVMObjectRValueRefArg:
       return "ObjectRValueRefArg";
+    case kTVMBFloat:
+      return "bf";

Review comment:
       ok, changed




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r439704765



##########
File path: tests/python/unittest/test_tir_transform_bf16_legalize.py
##########
@@ -0,0 +1,152 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import tvm
+import topi
+from tvm import te
+from tvm.tir import const
+
+
+def lower_stmt(sche, params, passfunc):
+    func = tvm.driver.build_module.form_irmodule(sche, params, "main", None)["main"]
+    func = passfunc()(
+        tvm.IRModule.from_expr(func))["main"]
+    stmt = func.body
+    return stmt
+
+
+def test_promote():
+    def runpass(op, passfunc):
+        a = te.placeholder((100,), dtype='bfloat16')
+        b = te.placeholder((100,), dtype='bfloat16')
+        c = te.compute((100,), lambda i: op(a[i], b[i]))
+        s = te.create_schedule(c.op)
+        return lower_stmt(s, [a, b, c], passfunc)
+    
+    def get_promoted(op):
+        a = te.placeholder((100,), dtype='bfloat16')
+        b = te.placeholder((100,), dtype='bfloat16')
+        c = te.compute((100,), lambda i:
+                topi.cast(op(topi.cast(a[i],'float'),
+                    topi.cast(b[i],'float')), 'bfloat16')
+                )
+        s = te.create_schedule(c.op)
+        func = tvm.driver.build_module.form_irmodule(s, [a,b,c], "main", None)["main"]
+        return func.body
+
+    def test_promoted(op):
+        stmt = runpass(op, tvm.tir.transform.BF16Promote)
+        tvm.ir.assert_structural_equal(stmt, get_promoted(op))
+    test_promoted(topi.add)
+    test_promoted(topi.subtract)
+    test_promoted(topi.multiply)
+    test_promoted(topi.divide)
+
+def test_eliminate():
+    def to32(v):
+        return topi.cast(v, 'float')
+    def to16(v):
+        return topi.cast(v, 'bfloat16')
+    def get_eliminated():
+        a = te.placeholder((100,), dtype='bfloat16')
+        b = te.placeholder((100,), dtype='bfloat16')
+        c = te.compute((100,), lambda i: to16(
+            topi.add(
+                to32(
+                    to16(
+                        topi.add(
+                            to32(a[i]),
+                            to32(b[i]),
+                        )
+                    )
+                ),
+                to32(
+                    to16(
+                        topi.add(
+                            to32(a[i]),
+                            to32(b[i]),
+                        )
+                    )
+                )
+            )
+        ))
+        s = te.create_schedule(c.op)
+        stmt = lower_stmt(s, [a, b, c], tvm.tir.transform.BF16CastElimination)
+        return stmt
+
+    def get_target():
+        a = te.placeholder((100,), dtype='bfloat16')
+        b = te.placeholder((100,), dtype='bfloat16')
+        c = te.compute((100,), lambda i: to16(
+            topi.add(topi.add(
+                        to32(a[i]),
+                        to32(b[i]),
+                    ),
+                    topi.add(
+                        to32(a[i]),
+                        to32(b[i]),
+                    )
+                )
+        ))
+        s = te.create_schedule(c.op)
+        func = tvm.driver.build_module.form_irmodule(s, [a,b,c], "main", None)["main"]
+        return func.body
+    tvm.ir.assert_structural_equal(get_eliminated(), get_target())
+
+def test_legalize():
+    def to32(v):
+        uint32_v = topi.cast(v, "uint32")
+        uint32_v = tvm.tir.call_pure_intrin("uint32", "shift_left", uint32_v, tvm.tir.const(16, "uint32"))
+        return tvm.tir.call_pure_intrin("float32", "reinterpret", uint32_v)
+    def to16(v):
+        uint32_v = tvm.tir.call_pure_intrin("uint32", "reinterpret", v)
+        rounding_bias = tvm.tir.call_pure_intrin("uint32", "shift_right", uint32_v, tvm.tir.const(16, "uint32"))
+        rounding_bias = tvm.tir.call_pure_intrin("uint32", "bitwise_and", rounding_bias, tvm.tir.const(1, "uint32"))
+        rounding_bias = rounding_bias + tvm.tir.const(0x7FFF, "uint16")
+        uint32_v = uint32_v + rounding_bias
+        uint32_v = tvm.tir.call_pure_intrin("uint32", "shift_right", uint32_v, tvm.tir.const(16, "uint32"))
+        return topi.cast(uint32_v, 'uint16')
+
+    def check(fcompute_before, fcompute_after):
+        a = te.placeholder((100,), dtype='bfloat16', name = 'A')
+        b = te.placeholder((100,), dtype='bfloat16', name = 'B')
+        c = te.compute((100,), fcompute_before(a,b), name = 'C')
+        s = te.create_schedule(c.op)
+        stmt = lower_stmt(s, [a, b, c], tvm.tir.transform.BF16Legalize)
+
+        a = te.placeholder((100,), dtype='uint16', name = 'A')
+        b = te.placeholder((100,), dtype='uint16', name = 'B')
+        c = te.compute((100,), fcompute_after(a,b), name = 'C')
+        s = te.create_schedule(c.op)
+        func = tvm.driver.build_module.form_irmodule(s, [a,b,c], "main", None)["main"]
+        tvm.ir.assert_structural_equal(stmt, func.body)
+
+    def orig1(a,b):
+        return lambda i: a[i]+b[i]+a[99-i]+b[99-i]
+    def after1(a,b):
+        return lambda i: to16(to32(a[i])+to32(b[i])+to32(a[99-i])+to32(b[99-i]))
+    def orig2(a,b):
+        return lambda i: a[i]*b[i]+a[99-i]*b[99-i]+a[i]
+    def after2(a,b):
+        return lambda i: to16(to32(a[i])*to32(b[i])+to32(a[99-i])*to32(b[99-i])+to32(a[i]))

Review comment:
       Sorry for that. Now I have formatted and manually run pylint on this test file. BTW, the test python files are never checked in TVM CI's pylint :)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen edited a comment on pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen edited a comment on pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#issuecomment-638583235


   So to summarize, we will need to introduce two kinds of TypeCode here. See the data type code defined here https://github.com/apache/incubator-tvm/blob/master/include/tvm/runtime/data_type.h#L48
   
   - DataTypeCode which is used by the DLDataType and in DLTensor
   - TVMTypeCode => TVMPackedArgTypeCode which is used by the packedfunc.
   
   I will try to raise a PR to do the split.
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r441249227



##########
File path: python/tvm/_ffi/runtime_ctypes.py
##########
@@ -96,6 +98,9 @@ def __init__(self, type_str):
             self.type_code = DataTypeCode.HANDLE
             bits = 64
             head = ""
+        elif head.startswith("bfloat"):
+            self.type_code = 4

Review comment:
       > not sure if it is good to hard code here
   
   Change to DataTypeCode. TVM refactors a lot (which is good). And when this PR was raised, all the type code here used hard codes.
   
   The other two issues you raised were also changed as required.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r427291901



##########
File path: src/target/llvm/codegen_llvm.cc
##########
@@ -309,6 +309,9 @@ llvm::Type* CodeGenLLVM::DTypeToLLVMType(const DataType& dtype) const {
       default:
         LOG(FATAL) << "do not support " << dtype;
     }
+  } else if (dtype.is_bfloat()) {
+    CHECK_EQ(dtype.bits(), 16);

Review comment:
       Sorry for the inclarity. I think in bfloat[X], only X=16 makes sense. But TVM's type system allows specifying the `bits` of a type. So here is the checking to make sure it is bf16.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r433016729



##########
File path: python/tvm/_ffi/runtime_ctypes.py
##########
@@ -58,7 +58,8 @@ class DataType(ctypes.Structure):
         0 : 'int',
         1 : 'uint',
         2 : 'float',
-        4 : 'handle'
+        4 : 'handle',
+        65: 'bfloat'

Review comment:
       @tqchen Hi, I found an inconsistency here: in Python binding, 4 is mapped to 'handle', while in C++ code, 4 is for Nullptr and 3 is for handle. Is this expected or a bug?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r427761508



##########
File path: include/tvm/runtime/c_runtime_api.h
##########
@@ -114,6 +114,7 @@ typedef enum {
   kTVMNNVMLast = 20U,
   // The following section of code is used for non-reserved types.
   kTVMExtReserveEnd = 64U,
+  kTVMBFloat = 65U,

Review comment:
       I suppose TVM should support kernel generation, e.g. generating a fused "conv+bn+relu", rather than generating end-to-end model, which is the usual case. In this case, we might select some intermediate layers of the model and let TVM generate the selected layers. The layers may require bf16 as the dtype, as they are in the middle of the model.
   
   What I want to say is that we sometimes need bf16 as the input dtype.  In our usecase in Intel, we need to generate a bf16 kernel (e.g. conv+bn+relu).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#issuecomment-646355311


   @junrushao1994 @ZhennanQin please followup and https://tvm.apache.org/docs/contribute/code_review.html#approve-and-request-changes-explicitly


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#issuecomment-641553734


   @vinx13 @ZihengJiang @liangfu it would be great if you cam take another look. Thanks @Menooker for keep improving the PR


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen merged pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen merged pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] junrushao1994 commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
junrushao1994 commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r442561558



##########
File path: include/tvm/runtime/data_type.h
##########
@@ -72,6 +73,9 @@ class DataType {
     data_.code = static_cast<uint8_t>(code);
     data_.bits = static_cast<uint8_t>(bits);
     data_.lanes = static_cast<uint16_t>(lanes);
+    if (code == kBFloat) {
+      CHECK_EQ(bits, 16);

Review comment:
       @tqchen This is just a nitpick. What do you think?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r427298389



##########
File path: src/target/llvm/codegen_llvm.cc
##########
@@ -906,7 +954,7 @@ DEFINE_CODEGEN_BINARY_OP(Mul);
   llvm::Value* CodeGenLLVM::Create##Op(DataType t, llvm::Value* a, llvm::Value* b) { \
     if (t.is_int()) {                                                                \
       return builder_->CreateICmpS##Op(a, b);                                        \
-    } else if (t.is_uint()) {                                                        \
+    } else if (t.is_uint() || t.is_bfloat()) {                                       \

Review comment:
       FP32/FP64 comparasion are also bit-wise in my understanding.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r429024767



##########
File path: src/target/llvm/codegen_llvm.cc
##########
@@ -555,12 +558,48 @@ void CodeGenLLVM::CreateSerialFor(llvm::Value* begin, llvm::Value* end, llvm::Va
   builder_->SetInsertPoint(for_end);
 }
 
+static llvm::Value* GetInt32VectorOrScalar(
+    llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>* builder, uint32_t v,
+    int lanes) {
+  if (lanes == 1) {
+    return builder->getInt32(v);
+  } else {
+    std::vector<llvm::Constant*> consts;
+    for (int i = 0; i < lanes; i++) {
+      consts.emplace_back(builder->getInt32(v));
+    }
+    return llvm::ConstantVector::get(consts);
+  }
+}
+
 // cast operatpr
 llvm::Value* CodeGenLLVM::CreateCast(DataType from, DataType to, llvm::Value* value) {
   llvm::Type* target = DTypeToLLVMType(to);
   if (value->getType() == target) return value;
   if (to.is_handle()) {
     return builder_->CreateBitCast(value, target);
+  } else if (to.is_float() && from.is_bfloat()) {
+    CHECK_EQ(from.bits(), 16);

Review comment:
       and I think the way processing “custom data type” that you mentioned does not fit this pr well. Actually I have already notice this feature before I wrote this bf16 feature. But it needs function calls to do lowering, which is not friendly to the codegen backend to do auto vectorization and so on. Of course you can say we can implement this cast function as an intrinsic. Yes, but more complexity is brought. 
   
   I think letting bf16 dtype live until codegen is a good idea, it makes legalization, impl of casting easier




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#issuecomment-643337519


   @junrushao1994 can you also take a quick look at this PR. thank you!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] liangfu commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
liangfu commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r427150166



##########
File path: src/target/llvm/codegen_llvm.cc
##########
@@ -309,6 +309,9 @@ llvm::Type* CodeGenLLVM::DTypeToLLVMType(const DataType& dtype) const {
       default:
         LOG(FATAL) << "do not support " << dtype;
     }
+  } else if (dtype.is_bfloat()) {
+    CHECK_EQ(dtype.bits(), 16);

Review comment:
       Since bfloat is assumed to be 16bit, can we keep the terminology more consistent? Since the data type is termed as `bf`, `bf16`, `bfloat16`, `bfloat` in the proposed change. Or are we going to support more data types like bfloat18 and bfloat20 in the future?

##########
File path: tests/python/unittest/test_tir_transform_bf16_legalize.py
##########
@@ -0,0 +1,139 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import tvm
+import topi
+from tvm import te
+from tvm.tir import const
+
+
+def lower_stmt(sche, params, passfunc):
+    func = tvm.driver.build_module.form_irmodule(sche, params, "main", None)["main"]
+    func = passfunc()(
+        tvm.IRModule.from_expr(func))["main"]
+    stmt = func.body
+    return stmt
+
+def to32(v):
+    return topi.cast(v, 'float')
+def to16(v):
+    return topi.cast(v, 'bf16')
+
+def test_promote():
+    def runpass(op, passfunc):
+        a = te.placeholder((100,), dtype='bf16')
+        b = te.placeholder((100,), dtype='bf16')
+        c = te.compute((100,), lambda i: op(a[i], b[i]))
+        s = te.create_schedule(c.op)
+        return lower_stmt(s, [a, b, c], passfunc)
+    
+    def get_promoted(op):
+        a = te.placeholder((100,), dtype='bf16')
+        b = te.placeholder((100,), dtype='bf16')
+        c = te.compute((100,), lambda i:
+                topi.cast(op(topi.cast(a[i],'float'),
+                    topi.cast(b[i],'float')), 'bf16')
+                )
+        s = te.create_schedule(c.op)
+        func = tvm.driver.build_module.form_irmodule(s, [a,b,c], "main", None)["main"]
+        return func.body
+
+    def test_promoted(op):
+        stmt = runpass(op, tvm.tir.transform.BF16Promote)
+        tvm.ir.assert_structural_equal(stmt, get_promoted(op))
+    test_promoted(topi.add)
+    test_promoted(topi.subtract)
+    test_promoted(topi.multiply)
+    test_promoted(topi.divide)
+
+def test_eliminate():
+    def get_eliminated():
+        a = te.placeholder((100,), dtype='bf16')
+        b = te.placeholder((100,), dtype='bf16')
+        c = te.compute((100,), lambda i: to16(
+            topi.add(
+                to32(
+                    to16(
+                        topi.add(
+                            to32(a[i]),
+                            to32(b[i]),
+                        )
+                    )
+                ),
+                to32(
+                    to16(
+                        topi.add(
+                            to32(a[i]),
+                            to32(b[i]),
+                        )
+                    )
+                )
+            )
+        ))
+        s = te.create_schedule(c.op)
+        stmt = lower_stmt(s, [a, b, c], tvm.tir.transform.BF16CastElimination)
+        return stmt
+
+    def get_target():
+        a = te.placeholder((100,), dtype='bf16')
+        b = te.placeholder((100,), dtype='bf16')
+        c = te.compute((100,), lambda i: to16(
+            topi.add(topi.add(
+                        to32(a[i]),
+                        to32(b[i]),
+                    ),
+                    topi.add(
+                        to32(a[i]),
+                        to32(b[i]),
+                    )
+                )
+        ))
+        s = te.create_schedule(c.op)
+        func = tvm.driver.build_module.form_irmodule(s, [a,b,c], "main", None)["main"]
+        return func.body
+
+    tvm.ir.assert_structural_equal(get_eliminated(), get_target())
+
+def test_legalize():
+    def check(fcompute_before, fcompute_after):
+        a = te.placeholder((100,), dtype='bf16')
+        b = te.placeholder((100,), dtype='bf16')
+        c = te.compute((100,), fcompute_before(a,b))
+        s = te.create_schedule(c.op)
+        stmt = lower_stmt(s, [a, b, c], tvm.tir.transform.BF16Legalize)
+
+        a = te.placeholder((100,), dtype='bf16')
+        b = te.placeholder((100,), dtype='bf16')
+        c = te.compute((100,), fcompute_after(a,b))
+        s = te.create_schedule(c.op)
+        func = tvm.driver.build_module.form_irmodule(s, [a,b,c], "main", None)["main"]
+        tvm.ir.assert_structural_equal(stmt, func.body)
+
+    def orig1(a,b):
+        return lambda i: a[i]+b[i]+a[99-i]+b[99-i]
+    def after1(a,b):
+        return lambda i: to16(to32(a[i])+to32(b[i])+to32(a[99-i])+to32(b[99-i]))
+    def orig1(a,b):
+        return lambda i: a[i]*b[i]+a[99-i]*b[99-i]+a[i]
+    def after1(a,b):
+        return lambda i: to16(to32(a[i])*to32(b[i])+to32(a[99-i])*to32(b[99-i])+to32(a[i]))
+
+    check(orig1, after1)
+
+if __name__ == "__main__":
+    test_promote()
+    test_eliminate()
+    test_legalize()

Review comment:
       Please leave a new line at EOF, even this is test script :)

##########
File path: src/target/llvm/codegen_llvm.cc
##########
@@ -906,7 +954,7 @@ DEFINE_CODEGEN_BINARY_OP(Mul);
   llvm::Value* CodeGenLLVM::Create##Op(DataType t, llvm::Value* a, llvm::Value* b) { \
     if (t.is_int()) {                                                                \
       return builder_->CreateICmpS##Op(a, b);                                        \
-    } else if (t.is_uint()) {                                                        \
+    } else if (t.is_uint() || t.is_bfloat()) {                                       \

Review comment:
       Isn't comparing bfloat16 this way risky?

##########
File path: src/target/llvm/codegen_llvm.cc
##########
@@ -555,12 +558,48 @@ void CodeGenLLVM::CreateSerialFor(llvm::Value* begin, llvm::Value* end, llvm::Va
   builder_->SetInsertPoint(for_end);
 }
 
+static llvm::Value* GetInt32VectorOrScalar(
+    llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>* builder, uint32_t v,
+    int lanes) {
+  if (lanes == 1) {
+    return builder->getInt32(v);
+  } else {
+    std::vector<llvm::Constant*> consts;
+    for (int i = 0; i < lanes; i++) {
+      consts.emplace_back(builder->getInt32(v));
+    }
+    return llvm::ConstantVector::get(consts);
+  }
+}
+
 // cast operatpr
 llvm::Value* CodeGenLLVM::CreateCast(DataType from, DataType to, llvm::Value* value) {
   llvm::Type* target = DTypeToLLVMType(to);
   if (value->getType() == target) return value;
   if (to.is_handle()) {
     return builder_->CreateBitCast(value, target);
+  } else if (to.is_float() && from.is_bfloat()) {
+    CHECK_EQ(from.bits(), 16);
+    CHECK_EQ(to.bits(), 32);
+    llvm::Type* extended_type = (from.lanes() == 1)
+                                    ? static_cast<llvm::Type*>(builder_->getInt32Ty())
+                                    : llvm::VectorType::get(builder_->getInt32Ty(), from.lanes());
+    auto v = builder_->CreateZExt(value, extended_type);
+    v = builder_->CreateShl(v, 16);

Review comment:
       Potential endianness problem here?

##########
File path: tests/python/unittest/test_target_codegen_llvm.py
##########
@@ -710,6 +710,52 @@ def _transform(f, *_):
         module(a_, b_, c_)
         tvm.testing.assert_allclose(c_.asnumpy(), (a_.asnumpy() * 2).astype('int32'))
 
+def np_float2np_bf16(arr):
+    ''' Convert a numpy array of float to a numpy array 
+    of bf16 in uint16'''
+    orig = arr.view('<u4')
+    bias = np.bitwise_and(np.right_shift(orig, 16), 1) + 0x7FFF
+    return np.right_shift(orig + bias, 16).astype('uint16')
+
+def np_float2tvm_bf16(arr):
+    ''' Convert a numpy array of float to a TVM array 
+    of bf16'''
+    nparr = np_float2np_bf16(arr)
+    return tvm.nd.empty(nparr.shape, 'bf16').copyfrom(nparr)
+
+def np_bf162np_float(arr):
+    ''' Convert a numpy array of bf16 (uint16) to a numpy array 
+    of float'''
+    u32 = np.left_shift(arr.astype('uint32'), 16)

Review comment:
       Are we going to produce a potential endianness problem here?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r427706238



##########
File path: include/tvm/runtime/c_runtime_api.h
##########
@@ -114,6 +114,7 @@ typedef enum {
   kTVMNNVMLast = 20U,
   // The following section of code is used for non-reserved types.
   kTVMExtReserveEnd = 64U,
+  kTVMBFloat = 65U,

Review comment:
       We do not want BFloat to be passed as PackedFunc argument, most packedfunc argument should always be passed as double

##########
File path: include/tvm/runtime/data_type.h
##########
@@ -297,6 +302,8 @@ inline const char* TypeCode2Str(int type_code) {
       return "Object";
     case kTVMObjectRValueRefArg:
       return "ObjectRValueRefArg";
+    case kTVMBFloat:
+      return "bf";

Review comment:
       bfloat

##########
File path: src/target/llvm/codegen_llvm.cc
##########
@@ -555,12 +558,48 @@ void CodeGenLLVM::CreateSerialFor(llvm::Value* begin, llvm::Value* end, llvm::Va
   builder_->SetInsertPoint(for_end);
 }
 
+static llvm::Value* GetInt32VectorOrScalar(
+    llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>* builder, uint32_t v,
+    int lanes) {
+  if (lanes == 1) {
+    return builder->getInt32(v);
+  } else {
+    std::vector<llvm::Constant*> consts;
+    for (int i = 0; i < lanes; i++) {
+      consts.emplace_back(builder->getInt32(v));
+    }
+    return llvm::ConstantVector::get(consts);
+  }
+}
+
 // cast operatpr
 llvm::Value* CodeGenLLVM::CreateCast(DataType from, DataType to, llvm::Value* value) {
   llvm::Type* target = DTypeToLLVMType(to);
   if (value->getType() == target) return value;
   if (to.is_handle()) {
     return builder_->CreateBitCast(value, target);
+  } else if (to.is_float() && from.is_bfloat()) {
+    CHECK_EQ(from.bits(), 16);

Review comment:
       If LLVM does not support bfloat, then perhaps we should do the legalization as a TIR=>TIR pass as opposed to do it in LLVM

##########
File path: include/tvm/runtime/data_type.h
##########
@@ -81,6 +82,10 @@ class DataType {
   bool is_float() const { return code() == DataType::kFloat; }
   /*! \return whether type is a float16 type. */
   bool is_float16() const { return is_float() && bits() == 16; }
+  /*! \return whether type is a bfloat type. */
+  bool is_bfloat() const { return code() == DataType::kBFloat; }

Review comment:
       given that only bfloat16 is defined, is_bf16 is a good enough function




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r434972874



##########
File path: include/tvm/runtime/data_type.h
##########
@@ -372,7 +372,7 @@ inline DLDataType String2DLDataType(std::string s) {
     t.lanes = 1;
     return t;
   } else if (s.substr(0, 6) == "bfloat") {
-    t.code = kTVMBFloat;
+    t.code = kDLBfloat;

Review comment:
       the type code should directly use DataType::kBfloat

##########
File path: python/tvm/_ffi/_cython/base.pxi
##########
@@ -27,7 +27,7 @@ cdef enum TVMTypeCode:
     kUInt = 1
     kFloat = 2
     kTVMOpaqueHandle = 3
-    kTVMNullptr = 4
+    kBFloat = 4

Review comment:
       sorry, we want to keep the type code for the normal argument as it is, and not changin the FFI




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#issuecomment-646674793


   Thanks @Menooker for being patient and keep improving the PR to maintain a high quality standard! Thanks @ZhennanQin @junrushao1994 @liangfu for helpful reviews!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r434955796



##########
File path: python/tvm/_ffi/runtime_ctypes.py
##########
@@ -58,7 +58,8 @@ class DataType(ctypes.Structure):
         0 : 'int',
         1 : 'uint',
         2 : 'float',
-        4 : 'handle'
+        4 : 'handle',
+        65: 'bfloat'

Review comment:
       4 is the type code of bfloat in dlpack




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r428114225



##########
File path: src/target/llvm/codegen_llvm.cc
##########
@@ -555,12 +558,48 @@ void CodeGenLLVM::CreateSerialFor(llvm::Value* begin, llvm::Value* end, llvm::Va
   builder_->SetInsertPoint(for_end);
 }
 
+static llvm::Value* GetInt32VectorOrScalar(
+    llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>* builder, uint32_t v,
+    int lanes) {
+  if (lanes == 1) {
+    return builder->getInt32(v);
+  } else {
+    std::vector<llvm::Constant*> consts;
+    for (int i = 0; i < lanes; i++) {
+      consts.emplace_back(builder->getInt32(v));
+    }
+    return llvm::ConstantVector::get(consts);
+  }
+}
+
 // cast operatpr
 llvm::Value* CodeGenLLVM::CreateCast(DataType from, DataType to, llvm::Value* value) {
   llvm::Type* target = DTypeToLLVMType(to);
   if (value->getType() == target) return value;
   if (to.is_handle()) {
     return builder_->CreateBitCast(value, target);
+  } else if (to.is_float() && from.is_bfloat()) {
+    CHECK_EQ(from.bits(), 16);

Review comment:
       Then we should directly change the type to be i16 during legalization and remove special handling code for bfloat16




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] junrushao1994 commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
junrushao1994 commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r439551002



##########
File path: tests/python/unittest/test_tir_transform_bf16_legalize.py
##########
@@ -0,0 +1,152 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import tvm
+import topi
+from tvm import te
+from tvm.tir import const
+
+
+def lower_stmt(sche, params, passfunc):
+    func = tvm.driver.build_module.form_irmodule(sche, params, "main", None)["main"]
+    func = passfunc()(
+        tvm.IRModule.from_expr(func))["main"]
+    stmt = func.body
+    return stmt
+
+
+def test_promote():
+    def runpass(op, passfunc):
+        a = te.placeholder((100,), dtype='bfloat16')
+        b = te.placeholder((100,), dtype='bfloat16')
+        c = te.compute((100,), lambda i: op(a[i], b[i]))
+        s = te.create_schedule(c.op)
+        return lower_stmt(s, [a, b, c], passfunc)
+    
+    def get_promoted(op):
+        a = te.placeholder((100,), dtype='bfloat16')
+        b = te.placeholder((100,), dtype='bfloat16')
+        c = te.compute((100,), lambda i:
+                topi.cast(op(topi.cast(a[i],'float'),
+                    topi.cast(b[i],'float')), 'bfloat16')
+                )
+        s = te.create_schedule(c.op)
+        func = tvm.driver.build_module.form_irmodule(s, [a,b,c], "main", None)["main"]
+        return func.body
+
+    def test_promoted(op):
+        stmt = runpass(op, tvm.tir.transform.BF16Promote)
+        tvm.ir.assert_structural_equal(stmt, get_promoted(op))
+    test_promoted(topi.add)
+    test_promoted(topi.subtract)
+    test_promoted(topi.multiply)
+    test_promoted(topi.divide)
+
+def test_eliminate():
+    def to32(v):
+        return topi.cast(v, 'float')
+    def to16(v):
+        return topi.cast(v, 'bfloat16')
+    def get_eliminated():
+        a = te.placeholder((100,), dtype='bfloat16')
+        b = te.placeholder((100,), dtype='bfloat16')
+        c = te.compute((100,), lambda i: to16(
+            topi.add(
+                to32(
+                    to16(
+                        topi.add(
+                            to32(a[i]),
+                            to32(b[i]),
+                        )
+                    )
+                ),
+                to32(
+                    to16(
+                        topi.add(
+                            to32(a[i]),
+                            to32(b[i]),
+                        )
+                    )
+                )
+            )
+        ))
+        s = te.create_schedule(c.op)
+        stmt = lower_stmt(s, [a, b, c], tvm.tir.transform.BF16CastElimination)
+        return stmt
+
+    def get_target():
+        a = te.placeholder((100,), dtype='bfloat16')
+        b = te.placeholder((100,), dtype='bfloat16')
+        c = te.compute((100,), lambda i: to16(
+            topi.add(topi.add(
+                        to32(a[i]),
+                        to32(b[i]),
+                    ),
+                    topi.add(
+                        to32(a[i]),
+                        to32(b[i]),
+                    )
+                )
+        ))
+        s = te.create_schedule(c.op)
+        func = tvm.driver.build_module.form_irmodule(s, [a,b,c], "main", None)["main"]
+        return func.body
+    tvm.ir.assert_structural_equal(get_eliminated(), get_target())
+
+def test_legalize():
+    def to32(v):
+        uint32_v = topi.cast(v, "uint32")
+        uint32_v = tvm.tir.call_pure_intrin("uint32", "shift_left", uint32_v, tvm.tir.const(16, "uint32"))
+        return tvm.tir.call_pure_intrin("float32", "reinterpret", uint32_v)
+    def to16(v):
+        uint32_v = tvm.tir.call_pure_intrin("uint32", "reinterpret", v)
+        rounding_bias = tvm.tir.call_pure_intrin("uint32", "shift_right", uint32_v, tvm.tir.const(16, "uint32"))
+        rounding_bias = tvm.tir.call_pure_intrin("uint32", "bitwise_and", rounding_bias, tvm.tir.const(1, "uint32"))
+        rounding_bias = rounding_bias + tvm.tir.const(0x7FFF, "uint16")
+        uint32_v = uint32_v + rounding_bias
+        uint32_v = tvm.tir.call_pure_intrin("uint32", "shift_right", uint32_v, tvm.tir.const(16, "uint32"))
+        return topi.cast(uint32_v, 'uint16')
+
+    def check(fcompute_before, fcompute_after):
+        a = te.placeholder((100,), dtype='bfloat16', name = 'A')
+        b = te.placeholder((100,), dtype='bfloat16', name = 'B')
+        c = te.compute((100,), fcompute_before(a,b), name = 'C')
+        s = te.create_schedule(c.op)
+        stmt = lower_stmt(s, [a, b, c], tvm.tir.transform.BF16Legalize)
+
+        a = te.placeholder((100,), dtype='uint16', name = 'A')
+        b = te.placeholder((100,), dtype='uint16', name = 'B')
+        c = te.compute((100,), fcompute_after(a,b), name = 'C')
+        s = te.create_schedule(c.op)
+        func = tvm.driver.build_module.form_irmodule(s, [a,b,c], "main", None)["main"]
+        tvm.ir.assert_structural_equal(stmt, func.body)
+
+    def orig1(a,b):
+        return lambda i: a[i]+b[i]+a[99-i]+b[99-i]
+    def after1(a,b):
+        return lambda i: to16(to32(a[i])+to32(b[i])+to32(a[99-i])+to32(b[99-i]))
+    def orig2(a,b):
+        return lambda i: a[i]*b[i]+a[99-i]*b[99-i]+a[i]
+    def after2(a,b):
+        return lambda i: to16(to32(a[i])*to32(b[i])+to32(a[99-i])*to32(b[99-i])+to32(a[i]))

Review comment:
       i am not so why the coding style here can pass pylint...Mind sending a simple fix?

##########
File path: tests/python/unittest/test_tir_transform_bf16_legalize.py
##########
@@ -0,0 +1,152 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import tvm
+import topi
+from tvm import te
+from tvm.tir import const
+
+
+def lower_stmt(sche, params, passfunc):
+    func = tvm.driver.build_module.form_irmodule(sche, params, "main", None)["main"]
+    func = passfunc()(
+        tvm.IRModule.from_expr(func))["main"]
+    stmt = func.body
+    return stmt
+
+
+def test_promote():
+    def runpass(op, passfunc):
+        a = te.placeholder((100,), dtype='bfloat16')
+        b = te.placeholder((100,), dtype='bfloat16')
+        c = te.compute((100,), lambda i: op(a[i], b[i]))
+        s = te.create_schedule(c.op)
+        return lower_stmt(s, [a, b, c], passfunc)
+    
+    def get_promoted(op):
+        a = te.placeholder((100,), dtype='bfloat16')
+        b = te.placeholder((100,), dtype='bfloat16')
+        c = te.compute((100,), lambda i:
+                topi.cast(op(topi.cast(a[i],'float'),
+                    topi.cast(b[i],'float')), 'bfloat16')
+                )
+        s = te.create_schedule(c.op)
+        func = tvm.driver.build_module.form_irmodule(s, [a,b,c], "main", None)["main"]
+        return func.body
+
+    def test_promoted(op):
+        stmt = runpass(op, tvm.tir.transform.BF16Promote)
+        tvm.ir.assert_structural_equal(stmt, get_promoted(op))
+    test_promoted(topi.add)
+    test_promoted(topi.subtract)
+    test_promoted(topi.multiply)
+    test_promoted(topi.divide)
+
+def test_eliminate():
+    def to32(v):
+        return topi.cast(v, 'float')
+    def to16(v):
+        return topi.cast(v, 'bfloat16')
+    def get_eliminated():
+        a = te.placeholder((100,), dtype='bfloat16')
+        b = te.placeholder((100,), dtype='bfloat16')
+        c = te.compute((100,), lambda i: to16(
+            topi.add(
+                to32(
+                    to16(
+                        topi.add(
+                            to32(a[i]),
+                            to32(b[i]),
+                        )
+                    )
+                ),
+                to32(
+                    to16(
+                        topi.add(
+                            to32(a[i]),
+                            to32(b[i]),
+                        )
+                    )
+                )
+            )
+        ))
+        s = te.create_schedule(c.op)
+        stmt = lower_stmt(s, [a, b, c], tvm.tir.transform.BF16CastElimination)
+        return stmt
+
+    def get_target():
+        a = te.placeholder((100,), dtype='bfloat16')
+        b = te.placeholder((100,), dtype='bfloat16')
+        c = te.compute((100,), lambda i: to16(
+            topi.add(topi.add(
+                        to32(a[i]),
+                        to32(b[i]),
+                    ),
+                    topi.add(
+                        to32(a[i]),
+                        to32(b[i]),
+                    )
+                )
+        ))
+        s = te.create_schedule(c.op)
+        func = tvm.driver.build_module.form_irmodule(s, [a,b,c], "main", None)["main"]
+        return func.body
+    tvm.ir.assert_structural_equal(get_eliminated(), get_target())
+
+def test_legalize():
+    def to32(v):
+        uint32_v = topi.cast(v, "uint32")
+        uint32_v = tvm.tir.call_pure_intrin("uint32", "shift_left", uint32_v, tvm.tir.const(16, "uint32"))
+        return tvm.tir.call_pure_intrin("float32", "reinterpret", uint32_v)
+    def to16(v):
+        uint32_v = tvm.tir.call_pure_intrin("uint32", "reinterpret", v)
+        rounding_bias = tvm.tir.call_pure_intrin("uint32", "shift_right", uint32_v, tvm.tir.const(16, "uint32"))
+        rounding_bias = tvm.tir.call_pure_intrin("uint32", "bitwise_and", rounding_bias, tvm.tir.const(1, "uint32"))
+        rounding_bias = rounding_bias + tvm.tir.const(0x7FFF, "uint16")
+        uint32_v = uint32_v + rounding_bias
+        uint32_v = tvm.tir.call_pure_intrin("uint32", "shift_right", uint32_v, tvm.tir.const(16, "uint32"))
+        return topi.cast(uint32_v, 'uint16')
+
+    def check(fcompute_before, fcompute_after):
+        a = te.placeholder((100,), dtype='bfloat16', name = 'A')
+        b = te.placeholder((100,), dtype='bfloat16', name = 'B')
+        c = te.compute((100,), fcompute_before(a,b), name = 'C')
+        s = te.create_schedule(c.op)
+        stmt = lower_stmt(s, [a, b, c], tvm.tir.transform.BF16Legalize)
+
+        a = te.placeholder((100,), dtype='uint16', name = 'A')
+        b = te.placeholder((100,), dtype='uint16', name = 'B')
+        c = te.compute((100,), fcompute_after(a,b), name = 'C')
+        s = te.create_schedule(c.op)
+        func = tvm.driver.build_module.form_irmodule(s, [a,b,c], "main", None)["main"]
+        tvm.ir.assert_structural_equal(stmt, func.body)
+
+    def orig1(a,b):
+        return lambda i: a[i]+b[i]+a[99-i]+b[99-i]
+    def after1(a,b):
+        return lambda i: to16(to32(a[i])+to32(b[i])+to32(a[99-i])+to32(b[99-i]))
+    def orig2(a,b):
+        return lambda i: a[i]*b[i]+a[99-i]*b[99-i]+a[i]
+    def after2(a,b):
+        return lambda i: to16(to32(a[i])*to32(b[i])+to32(a[99-i])*to32(b[99-i])+to32(a[i]))

Review comment:
       i am not so sure why the coding style here can pass pylint...Mind sending a simple fix?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#issuecomment-638583235


   See the data type code defined here https://github.com/apache/incubator-tvm/blob/master/include/tvm/runtime/data_type.h#L48


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r428126396



##########
File path: include/tvm/runtime/c_runtime_api.h
##########
@@ -114,6 +114,7 @@ typedef enum {
   kTVMNNVMLast = 20U,
   // The following section of code is used for non-reserved types.
   kTVMExtReserveEnd = 64U,
+  kTVMBFloat = 65U,

Review comment:
       The particular code is used when we directly pass a constant into PackedFunc, e.g. `f(1.0, some_float_value)`. in these cases double can be used.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r441249227



##########
File path: python/tvm/_ffi/runtime_ctypes.py
##########
@@ -96,6 +98,9 @@ def __init__(self, type_str):
             self.type_code = DataTypeCode.HANDLE
             bits = 64
             head = ""
+        elif head.startswith("bfloat"):
+            self.type_code = 4

Review comment:
       > not sure if it is good to hard code here
   
   Change to DataTypeCode. TVM refactors a lot (which is good). And when this PR is raised, all the type code here uses hard codes.
   
   The other two issues you raised are also changed as required.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] liangfu commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
liangfu commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r427691578



##########
File path: tests/python/unittest/test_target_codegen_llvm.py
##########
@@ -710,6 +710,52 @@ def _transform(f, *_):
         module(a_, b_, c_)
         tvm.testing.assert_allclose(c_.asnumpy(), (a_.asnumpy() * 2).astype('int32'))
 
+def np_float2np_bf16(arr):
+    ''' Convert a numpy array of float to a numpy array 
+    of bf16 in uint16'''
+    orig = arr.view('<u4')
+    bias = np.bitwise_and(np.right_shift(orig, 16), 1) + 0x7FFF
+    return np.right_shift(orig + bias, 16).astype('uint16')
+
+def np_float2tvm_bf16(arr):
+    ''' Convert a numpy array of float to a TVM array 
+    of bf16'''
+    nparr = np_float2np_bf16(arr)
+    return tvm.nd.empty(nparr.shape, 'bf16').copyfrom(nparr)
+
+def np_bf162np_float(arr):
+    ''' Convert a numpy array of bf16 (uint16) to a numpy array 
+    of float'''
+    u32 = np.left_shift(arr.astype('uint32'), 16)

Review comment:
       Can we reused the following code snippet, which preserves endianness checks?
   
   https://github.com/apache/incubator-tvm/blob/6cbda80227fc18a859c4b01f57f75abbd7a16181/3rdparty/bfloat16/bfloat16.cc#L27-L35




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r432449575



##########
File path: src/tir/transforms/bf16_legalize.cc
##########
@@ -0,0 +1,384 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file bf16_legalize.cc
+ * \brief legalize bf16 type by adding cast_to_fp32
+ */
+
+#include <tvm/runtime/registry.h>
+#include <tvm/tir/op.h>
+#include <tvm/tir/transform.h>
+
+#include <cmath>
+#include <tuple>
+
+#include "../../arith/ir_mutator_with_analyzer.h"
+#include "../../arith/ir_visitor_with_analyzer.h"
+
+namespace tvm {
+namespace tir {
+
+using arith::Analyzer;
+using arith::IRMutatorWithAnalyzer;
+
+class BF16PromoteRewriter : public StmtExprMutator {
+ public:
+  BF16PromoteRewriter() {}
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  std::tuple<PrimExpr, PrimExpr> DoCast(PrimExpr orig_a, PrimExpr orig_b, bool* is_bfloat16) {
+    auto a = this->VisitExpr(orig_a);
+    auto b = this->VisitExpr(orig_b);
+    *is_bfloat16 = false;
+    if (a->dtype.is_bfloat16()) {
+      CHECK(b->dtype.is_bfloat16());
+      *is_bfloat16 = true;
+    } else if (b->dtype.is_bfloat16()) {
+      CHECK(a->dtype.is_bfloat16());
+      *is_bfloat16 = true;
+    }
+
+    if (*is_bfloat16) {
+      DataType fp32ty(kDLFloat, 32, 1);
+      a = CastNode::make(fp32ty, a);
+      b = CastNode::make(fp32ty, b);
+    }
+    return std::make_tuple(a, b);
+  }
+
+  PrimExpr VisitExpr_(const AddNode* op) final;
+  PrimExpr VisitExpr_(const SubNode* op) final;
+  PrimExpr VisitExpr_(const MulNode* op) final;
+  PrimExpr VisitExpr_(const DivNode* op) final;
+  PrimExpr VisitExpr_(const MinNode* op) final;
+  PrimExpr VisitExpr_(const MaxNode* op) final;
+  PrimExpr VisitExpr_(const LTNode* op) final;
+  PrimExpr VisitExpr_(const LENode* op) final;
+  PrimExpr VisitExpr_(const GTNode* op) final;
+  PrimExpr VisitExpr_(const GENode* op) final;
+};
+
+#define DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(OP, FUNC)        \
+  PrimExpr BF16PromoteRewriter::VisitExpr_(const OP* op) {       \
+    PrimExpr a, b;                                               \
+    bool is_bfloat16;                                            \
+    std::tie(a, b) = DoCast(op->a, op->b, &is_bfloat16);         \
+    if (a.same_as(op->a) && b.same_as(op->b)) {                  \
+      return GetRef<PrimExpr>(op);                               \
+    } else {                                                     \
+      auto ret = FUNC(a, b);                                     \
+      if (!is_bfloat16)                                          \
+        return ret;                                              \
+      else                                                       \
+        return CastNode::make(DataType(kTVMBFloat, 16, 1), ret); \
+    }                                                            \
+  }
+
+#define DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(OP, FUNC) \
+  PrimExpr BF16PromoteRewriter::VisitExpr_(const OP* op) {        \
+    PrimExpr a, b;                                                \
+    bool is_bfloat16;                                             \
+    std::tie(a, b) = DoCast(op->a, op->b, &is_bfloat16);          \
+    if (a.same_as(op->a) && b.same_as(op->b)) {                   \
+      return GetRef<PrimExpr>(op);                                \
+    } else {                                                      \
+      auto ret = FUNC(a, b);                                      \
+      return ret;                                                 \
+    }                                                             \
+  }
+
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(AddNode, operator+)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(SubNode, operator-)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MulNode, operator*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(DivNode, div)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MinNode, min)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MaxNode, max)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(LTNode, operator<)   // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(LENode, operator<=)  // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(GTNode, operator>)   // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(GENode, operator>=)  // NOLINT(*)
+
+/*
+ * Eliminate verbose casting between fp32 and bf16
+ * Checks if the AST has the pattern:
+ *     castto32(castto16(some_fp32_op(...)))
+ * The verbose casting is generated by BF16Promote for multiple
+ * bf16 Ops in a row. e.g.:
+ *  X[i] + Y[i] + T[i] =>
+ *  bf16((float32(bf16((float32(X[i]) + float32(Y[i])))) + float32(T[i])))
+ * After this pass:
+ *  bf16(float32(X[i]) + float32(Y[i]) + float32(T[i]))
+ */
+class BF16CastEliminationRewriter : public StmtExprMutator {
+ public:
+  BF16CastEliminationRewriter() {}
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  PrimExpr VisitExpr_(const CastNode* op) final {
+    auto op_val = StmtExprMutator::VisitExpr(op->value);
+    if (op->dtype.is_float() && op->dtype.bits() == 32) {
+      // if is cast_to_fp32, check if op->value is cast_to_fp16
+      // and op->value->value is a float32
+      if (auto innercast = op_val.as<CastNode>()) {
+        if (innercast->dtype.is_bfloat16() && innercast->value->dtype.is_float() &&
+            innercast->value->dtype.bits() == 32) {
+          return innercast->value;
+        }
+      }
+    }
+    if (op->value.same_as(op_val)) return GetRef<PrimExpr>(op);
+    return CastNode::make(op->dtype, op_val);
+  }
+};
+
+// implementation from
+// https://github.com/pytorch/pytorch/blob/master/c10/util/BFloat16.h
+inline uint16_t round_to_nearest_even(float src) {
+  if (std::isnan(src)) {
+    return UINT16_C(0x7FC0);
+  } else {
+    union {
+      uint32_t U32;
+      float F32;
+    };
+
+    F32 = src;
+    uint32_t rounding_bias = ((U32 >> 16) & 1) + UINT32_C(0x7FFF);
+    return static_cast<uint16_t>((U32 + rounding_bias) >> 16);
+  }
+}
+
+/*
+ * Lower the bf16 type to int16
+ * Lower cast between bf16 and fp32
+ * Lower bf16 FloatImm to int16
+ */
+class BF16LowerRewriter : StmtExprMutator {
+ public:
+  BF16LowerRewriter() {}
+
+  std::unordered_map<const BufferNode*, Buffer> buffer_remap;
+  std::unordered_map<const VarNode*, Var> var_remap;
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  PrimExpr VisitExpr_(const CastNode* op) final {
+    auto op_val = StmtExprMutator::VisitExpr(op->value);
+    if (op->value->dtype.is_bfloat16()) {
+      // if is cast_from_bf16, check if is to fp32
+      CHECK(op->dtype.is_float() && op->dtype.bits() == 32);
+      auto uint32_dtype = DataType(kDLUInt, 32, op_val->dtype.lanes());
+      auto uint32_v = CastNode::make(uint32_dtype, op_val);
+      return CallNode::make(op->dtype, CallNode::reinterpret, {uint32_v << 16},
+                            CallNode::PureIntrinsic);
+
+    } else if (op->dtype.is_bfloat16()) {
+      // if is cast_to_bf16, check if op->value is fp32
+      CHECK(op->value->dtype.is_float() && op->value->dtype.bits() == 32);
+      auto uint32_dtype = DataType(kDLUInt, 32, op_val->dtype.lanes());
+      auto uint32_v =
+          CallNode::make(uint32_dtype, CallNode::reinterpret, {op_val}, CallNode::PureIntrinsic);
+      auto uint16_dtype = DataType(kDLUInt, 16, op_val->dtype.lanes());
+      // uint32_t rounding_bias = ((U32 >> 16) & 1) + UINT32_C(0x7FFF);
+      // return static_cast<uint16_t>((U32 + rounding_bias) >> 16);
+      auto rounding_bias = ((uint32_v >> 16) & 1) + make_const(uint16_dtype, 0x7FFF);
+      return CastNode::make(uint16_dtype, {(uint32_v + rounding_bias) >> 16});

Review comment:
       ok, done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r427306003



##########
File path: tests/python/unittest/test_target_codegen_llvm.py
##########
@@ -710,6 +710,52 @@ def _transform(f, *_):
         module(a_, b_, c_)
         tvm.testing.assert_allclose(c_.asnumpy(), (a_.asnumpy() * 2).astype('int32'))
 
+def np_float2np_bf16(arr):
+    ''' Convert a numpy array of float to a numpy array 
+    of bf16 in uint16'''
+    orig = arr.view('<u4')
+    bias = np.bitwise_and(np.right_shift(orig, 16), 1) + 0x7FFF
+    return np.right_shift(orig + bias, 16).astype('uint16')
+
+def np_float2tvm_bf16(arr):
+    ''' Convert a numpy array of float to a TVM array 
+    of bf16'''
+    nparr = np_float2np_bf16(arr)
+    return tvm.nd.empty(nparr.shape, 'bf16').copyfrom(nparr)
+
+def np_bf162np_float(arr):
+    ''' Convert a numpy array of bf16 (uint16) to a numpy array 
+    of float'''
+    u32 = np.left_shift(arr.astype('uint32'), 16)

Review comment:
       In my understanding, fp32=>bf16 casting preserves the higher-ordered bits (bits 31-16). We don't need to know whether the higher-ordered bits are stored in a larger address or a smaller address (which is the endianness), we just need to get the bits by shifting, which is well-defined - just using shifting is enough.  
   
   Reference: [wiki for fp32 bit order](https://en.wikipedia.org/wiki/Single-precision_floating-point_format)
   
   [PyTorch's bf16 casting](https://github.com/pytorch/pytorch/blob/master/c10/util/BFloat16.h)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r427306003



##########
File path: tests/python/unittest/test_target_codegen_llvm.py
##########
@@ -710,6 +710,52 @@ def _transform(f, *_):
         module(a_, b_, c_)
         tvm.testing.assert_allclose(c_.asnumpy(), (a_.asnumpy() * 2).astype('int32'))
 
+def np_float2np_bf16(arr):
+    ''' Convert a numpy array of float to a numpy array 
+    of bf16 in uint16'''
+    orig = arr.view('<u4')
+    bias = np.bitwise_and(np.right_shift(orig, 16), 1) + 0x7FFF
+    return np.right_shift(orig + bias, 16).astype('uint16')
+
+def np_float2tvm_bf16(arr):
+    ''' Convert a numpy array of float to a TVM array 
+    of bf16'''
+    nparr = np_float2np_bf16(arr)
+    return tvm.nd.empty(nparr.shape, 'bf16').copyfrom(nparr)
+
+def np_bf162np_float(arr):
+    ''' Convert a numpy array of bf16 (uint16) to a numpy array 
+    of float'''
+    u32 = np.left_shift(arr.astype('uint32'), 16)

Review comment:
       In my understanding, fp32=>bf16 casting preserves the higher-ordered bits (bits 31-16). We don't need to know whether the higher-ordered bits are stored in a larger address or a smaller address (which is the endianness), we just need to get the bits by shifting, which is well-defined - just using shifting is enough.  
   
   Reference: [wiki for fp32 bit order](https://en.wikipedia.org/wiki/Single-precision_floating-point_format)
   
   [PyTorch's bf16 casting] (https://github.com/pytorch/pytorch/blob/master/c10/util/BFloat16.h)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#issuecomment-638582535


   I want to clarify the case of the TypeCode. Given that we are not trying to making a consistent type code for arguments beyond kOpaqueHandle.
   
   So in this case, we shall simply keep the 4 for kTVMNullPtr code, not changing the FFI, and separate DataTypeCode and Argument type code.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] ZhennanQin commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
ZhennanQin commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r432418422



##########
File path: src/tir/transforms/bf16_legalize.cc
##########
@@ -0,0 +1,384 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file bf16_legalize.cc
+ * \brief legalize bf16 type by adding cast_to_fp32
+ */
+
+#include <tvm/runtime/registry.h>
+#include <tvm/tir/op.h>
+#include <tvm/tir/transform.h>
+
+#include <cmath>
+#include <tuple>
+
+#include "../../arith/ir_mutator_with_analyzer.h"
+#include "../../arith/ir_visitor_with_analyzer.h"
+
+namespace tvm {
+namespace tir {
+
+using arith::Analyzer;
+using arith::IRMutatorWithAnalyzer;
+
+class BF16PromoteRewriter : public StmtExprMutator {
+ public:
+  BF16PromoteRewriter() {}
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  std::tuple<PrimExpr, PrimExpr> DoCast(PrimExpr orig_a, PrimExpr orig_b, bool* is_bfloat16) {
+    auto a = this->VisitExpr(orig_a);
+    auto b = this->VisitExpr(orig_b);
+    *is_bfloat16 = false;
+    if (a->dtype.is_bfloat16()) {
+      CHECK(b->dtype.is_bfloat16());
+      *is_bfloat16 = true;
+    } else if (b->dtype.is_bfloat16()) {
+      CHECK(a->dtype.is_bfloat16());
+      *is_bfloat16 = true;
+    }
+
+    if (*is_bfloat16) {
+      DataType fp32ty(kDLFloat, 32, 1);
+      a = CastNode::make(fp32ty, a);
+      b = CastNode::make(fp32ty, b);
+    }
+    return std::make_tuple(a, b);
+  }
+
+  PrimExpr VisitExpr_(const AddNode* op) final;
+  PrimExpr VisitExpr_(const SubNode* op) final;
+  PrimExpr VisitExpr_(const MulNode* op) final;
+  PrimExpr VisitExpr_(const DivNode* op) final;
+  PrimExpr VisitExpr_(const MinNode* op) final;
+  PrimExpr VisitExpr_(const MaxNode* op) final;
+  PrimExpr VisitExpr_(const LTNode* op) final;
+  PrimExpr VisitExpr_(const LENode* op) final;
+  PrimExpr VisitExpr_(const GTNode* op) final;
+  PrimExpr VisitExpr_(const GENode* op) final;
+};
+
+#define DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(OP, FUNC)        \
+  PrimExpr BF16PromoteRewriter::VisitExpr_(const OP* op) {       \
+    PrimExpr a, b;                                               \
+    bool is_bfloat16;                                            \
+    std::tie(a, b) = DoCast(op->a, op->b, &is_bfloat16);         \
+    if (a.same_as(op->a) && b.same_as(op->b)) {                  \
+      return GetRef<PrimExpr>(op);                               \
+    } else {                                                     \
+      auto ret = FUNC(a, b);                                     \
+      if (!is_bfloat16)                                          \
+        return ret;                                              \
+      else                                                       \
+        return CastNode::make(DataType(kTVMBFloat, 16, 1), ret); \
+    }                                                            \
+  }
+
+#define DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(OP, FUNC) \
+  PrimExpr BF16PromoteRewriter::VisitExpr_(const OP* op) {        \
+    PrimExpr a, b;                                                \
+    bool is_bfloat16;                                             \
+    std::tie(a, b) = DoCast(op->a, op->b, &is_bfloat16);          \
+    if (a.same_as(op->a) && b.same_as(op->b)) {                   \
+      return GetRef<PrimExpr>(op);                                \
+    } else {                                                      \
+      auto ret = FUNC(a, b);                                      \
+      return ret;                                                 \
+    }                                                             \
+  }
+
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(AddNode, operator+)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(SubNode, operator-)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MulNode, operator*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(DivNode, div)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MinNode, min)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MaxNode, max)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(LTNode, operator<)   // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(LENode, operator<=)  // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(GTNode, operator>)   // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(GENode, operator>=)  // NOLINT(*)
+
+/*
+ * Eliminate verbose casting between fp32 and bf16
+ * Checks if the AST has the pattern:
+ *     castto32(castto16(some_fp32_op(...)))
+ * The verbose casting is generated by BF16Promote for multiple
+ * bf16 Ops in a row. e.g.:
+ *  X[i] + Y[i] + T[i] =>
+ *  bf16((float32(bf16((float32(X[i]) + float32(Y[i])))) + float32(T[i])))
+ * After this pass:
+ *  bf16(float32(X[i]) + float32(Y[i]) + float32(T[i]))
+ */
+class BF16CastEliminationRewriter : public StmtExprMutator {
+ public:
+  BF16CastEliminationRewriter() {}
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  PrimExpr VisitExpr_(const CastNode* op) final {
+    auto op_val = StmtExprMutator::VisitExpr(op->value);
+    if (op->dtype.is_float() && op->dtype.bits() == 32) {
+      // if is cast_to_fp32, check if op->value is cast_to_fp16
+      // and op->value->value is a float32
+      if (auto innercast = op_val.as<CastNode>()) {
+        if (innercast->dtype.is_bfloat16() && innercast->value->dtype.is_float() &&
+            innercast->value->dtype.bits() == 32) {
+          return innercast->value;
+        }
+      }
+    }
+    if (op->value.same_as(op_val)) return GetRef<PrimExpr>(op);
+    return CastNode::make(op->dtype, op_val);
+  }
+};
+
+// implementation from
+// https://github.com/pytorch/pytorch/blob/master/c10/util/BFloat16.h
+inline uint16_t round_to_nearest_even(float src) {
+  if (std::isnan(src)) {
+    return UINT16_C(0x7FC0);
+  } else {
+    union {
+      uint32_t U32;
+      float F32;
+    };
+
+    F32 = src;
+    uint32_t rounding_bias = ((U32 >> 16) & 1) + UINT32_C(0x7FFF);
+    return static_cast<uint16_t>((U32 + rounding_bias) >> 16);
+  }
+}
+
+/*
+ * Lower the bf16 type to int16
+ * Lower cast between bf16 and fp32
+ * Lower bf16 FloatImm to int16
+ */
+class BF16LowerRewriter : StmtExprMutator {
+ public:
+  BF16LowerRewriter() {}
+
+  std::unordered_map<const BufferNode*, Buffer> buffer_remap;
+  std::unordered_map<const VarNode*, Var> var_remap;
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  PrimExpr VisitExpr_(const CastNode* op) final {
+    auto op_val = StmtExprMutator::VisitExpr(op->value);
+    if (op->value->dtype.is_bfloat16()) {
+      // if is cast_from_bf16, check if is to fp32
+      CHECK(op->dtype.is_float() && op->dtype.bits() == 32);
+      auto uint32_dtype = DataType(kDLUInt, 32, op_val->dtype.lanes());
+      auto uint32_v = CastNode::make(uint32_dtype, op_val);
+      return CallNode::make(op->dtype, CallNode::reinterpret, {uint32_v << 16},
+                            CallNode::PureIntrinsic);
+
+    } else if (op->dtype.is_bfloat16()) {
+      // if is cast_to_bf16, check if op->value is fp32
+      CHECK(op->value->dtype.is_float() && op->value->dtype.bits() == 32);
+      auto uint32_dtype = DataType(kDLUInt, 32, op_val->dtype.lanes());
+      auto uint32_v =
+          CallNode::make(uint32_dtype, CallNode::reinterpret, {op_val}, CallNode::PureIntrinsic);
+      auto uint16_dtype = DataType(kDLUInt, 16, op_val->dtype.lanes());
+      // uint32_t rounding_bias = ((U32 >> 16) & 1) + UINT32_C(0x7FFF);

Review comment:
       @Menooker Then you'd better re-format this line as a comment. Like line 120-128




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] junrushao1994 commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
junrushao1994 commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r441191343



##########
File path: include/tvm/runtime/data_type.h
##########
@@ -72,6 +73,9 @@ class DataType {
     data_.code = static_cast<uint8_t>(code);
     data_.bits = static_cast<uint8_t>(bits);
     data_.lanes = static_cast<uint16_t>(lanes);
+    if (code == kBFloat) {
+      CHECK_EQ(bits, 16);

Review comment:
       It is understandable that right now we only support bf16, but my concern is that "should we put the check here"?

##########
File path: include/tvm/runtime/data_type.h
##########
@@ -372,7 +372,7 @@ inline DLDataType String2DLDataType(std::string s) {
     t.lanes = 1;
     return t;
   } else if (s.substr(0, 6) == "bfloat") {
-    t.code = kTVMBFloat;
+    t.code = kDLBfloat;

Review comment:
       I agree with tq

##########
File path: python/tvm/_ffi/_cython/base.pxi
##########
@@ -27,7 +27,7 @@ cdef enum TVMTypeCode:
     kUInt = 1
     kFloat = 2
     kTVMOpaqueHandle = 3
-    kTVMNullptr = 4
+    kBFloat = 4

Review comment:
       shall we remove this?

##########
File path: python/tvm/_ffi/runtime_ctypes.py
##########
@@ -96,6 +98,9 @@ def __init__(self, type_str):
             self.type_code = DataTypeCode.HANDLE
             bits = 64
             head = ""
+        elif head.startswith("bfloat"):
+            self.type_code = 4

Review comment:
       not sure if it is good to hard code here




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r428427694



##########
File path: include/tvm/runtime/c_runtime_api.h
##########
@@ -114,6 +114,7 @@ typedef enum {
   kTVMNNVMLast = 20U,
   // The following section of code is used for non-reserved types.
   kTVMExtReserveEnd = 64U,
+  kTVMBFloat = 65U,

Review comment:
       If we remove this type from TVM runtime, we cannot pass a bf16 array to TVM via Python and users can only pass bf16 buffers via C runtime (or in some awkward way to construct a bf16 DLTensor via Python). Currently, with  kTVMBFloat defined, we can:
   
   ```python
           A = te.placeholder((32, ), dtype='bfloat16')
           B = te.placeholder((32, ), dtype='bfloat16')
           d = te.compute((32, ), lambda x: A[x] + B[x])
           sch = te.create_schedule(d.op)
           module = tvm.build(sch, [A, B, d])
           npa = np.random.rand(32).astype('float32')
           npb = np.random.rand(32).astype('float32')
           a_ = np_float2tvm_bf16(npa)
           b_ = np_float2tvm_bf16(npb)
           c_ = tvm.nd.empty((32,), 'bfloat16')
           module(a_, b_, c_)
   ```
   
   Which is useful for testing and prototyping.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] liangfu commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
liangfu commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r432266454



##########
File path: src/tir/transforms/bf16_legalize.cc
##########
@@ -0,0 +1,384 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file bf16_legalize.cc
+ * \brief legalize bf16 type by adding cast_to_fp32
+ */
+
+#include <tvm/runtime/registry.h>
+#include <tvm/tir/op.h>
+#include <tvm/tir/transform.h>
+
+#include <cmath>
+#include <tuple>
+
+#include "../../arith/ir_mutator_with_analyzer.h"
+#include "../../arith/ir_visitor_with_analyzer.h"
+
+namespace tvm {
+namespace tir {
+
+using arith::Analyzer;
+using arith::IRMutatorWithAnalyzer;
+
+class BF16PromoteRewriter : public StmtExprMutator {
+ public:
+  BF16PromoteRewriter() {}
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  std::tuple<PrimExpr, PrimExpr> DoCast(PrimExpr orig_a, PrimExpr orig_b, bool* is_bfloat16) {
+    auto a = this->VisitExpr(orig_a);
+    auto b = this->VisitExpr(orig_b);
+    *is_bfloat16 = false;
+    if (a->dtype.is_bfloat16()) {
+      CHECK(b->dtype.is_bfloat16());
+      *is_bfloat16 = true;
+    } else if (b->dtype.is_bfloat16()) {
+      CHECK(a->dtype.is_bfloat16());
+      *is_bfloat16 = true;
+    }
+
+    if (*is_bfloat16) {
+      DataType fp32ty(kDLFloat, 32, 1);
+      a = CastNode::make(fp32ty, a);
+      b = CastNode::make(fp32ty, b);
+    }
+    return std::make_tuple(a, b);
+  }
+
+  PrimExpr VisitExpr_(const AddNode* op) final;
+  PrimExpr VisitExpr_(const SubNode* op) final;
+  PrimExpr VisitExpr_(const MulNode* op) final;
+  PrimExpr VisitExpr_(const DivNode* op) final;
+  PrimExpr VisitExpr_(const MinNode* op) final;
+  PrimExpr VisitExpr_(const MaxNode* op) final;
+  PrimExpr VisitExpr_(const LTNode* op) final;
+  PrimExpr VisitExpr_(const LENode* op) final;
+  PrimExpr VisitExpr_(const GTNode* op) final;
+  PrimExpr VisitExpr_(const GENode* op) final;
+};
+
+#define DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(OP, FUNC)        \
+  PrimExpr BF16PromoteRewriter::VisitExpr_(const OP* op) {       \
+    PrimExpr a, b;                                               \
+    bool is_bfloat16;                                            \
+    std::tie(a, b) = DoCast(op->a, op->b, &is_bfloat16);         \
+    if (a.same_as(op->a) && b.same_as(op->b)) {                  \
+      return GetRef<PrimExpr>(op);                               \
+    } else {                                                     \
+      auto ret = FUNC(a, b);                                     \
+      if (!is_bfloat16)                                          \
+        return ret;                                              \
+      else                                                       \
+        return CastNode::make(DataType(kTVMBFloat, 16, 1), ret); \
+    }                                                            \
+  }
+
+#define DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(OP, FUNC) \
+  PrimExpr BF16PromoteRewriter::VisitExpr_(const OP* op) {        \
+    PrimExpr a, b;                                                \
+    bool is_bfloat16;                                             \
+    std::tie(a, b) = DoCast(op->a, op->b, &is_bfloat16);          \
+    if (a.same_as(op->a) && b.same_as(op->b)) {                   \
+      return GetRef<PrimExpr>(op);                                \
+    } else {                                                      \
+      auto ret = FUNC(a, b);                                      \
+      return ret;                                                 \
+    }                                                             \
+  }
+
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(AddNode, operator+)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(SubNode, operator-)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MulNode, operator*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(DivNode, div)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MinNode, min)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MaxNode, max)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(LTNode, operator<)   // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(LENode, operator<=)  // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(GTNode, operator>)   // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(GENode, operator>=)  // NOLINT(*)
+
+/*
+ * Eliminate verbose casting between fp32 and bf16
+ * Checks if the AST has the pattern:
+ *     castto32(castto16(some_fp32_op(...)))
+ * The verbose casting is generated by BF16Promote for multiple
+ * bf16 Ops in a row. e.g.:
+ *  X[i] + Y[i] + T[i] =>
+ *  bf16((float32(bf16((float32(X[i]) + float32(Y[i])))) + float32(T[i])))
+ * After this pass:
+ *  bf16(float32(X[i]) + float32(Y[i]) + float32(T[i]))
+ */
+class BF16CastEliminationRewriter : public StmtExprMutator {
+ public:
+  BF16CastEliminationRewriter() {}
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  PrimExpr VisitExpr_(const CastNode* op) final {
+    auto op_val = StmtExprMutator::VisitExpr(op->value);
+    if (op->dtype.is_float() && op->dtype.bits() == 32) {
+      // if is cast_to_fp32, check if op->value is cast_to_fp16
+      // and op->value->value is a float32
+      if (auto innercast = op_val.as<CastNode>()) {
+        if (innercast->dtype.is_bfloat16() && innercast->value->dtype.is_float() &&
+            innercast->value->dtype.bits() == 32) {
+          return innercast->value;
+        }
+      }
+    }
+    if (op->value.same_as(op_val)) return GetRef<PrimExpr>(op);
+    return CastNode::make(op->dtype, op_val);
+  }
+};
+
+// implementation from
+// https://github.com/pytorch/pytorch/blob/master/c10/util/BFloat16.h
+inline uint16_t round_to_nearest_even(float src) {
+  if (std::isnan(src)) {
+    return UINT16_C(0x7FC0);
+  } else {
+    union {
+      uint32_t U32;
+      float F32;
+    };
+
+    F32 = src;
+    uint32_t rounding_bias = ((U32 >> 16) & 1) + UINT32_C(0x7FFF);
+    return static_cast<uint16_t>((U32 + rounding_bias) >> 16);
+  }
+}
+
+/*
+ * Lower the bf16 type to int16
+ * Lower cast between bf16 and fp32
+ * Lower bf16 FloatImm to int16
+ */
+class BF16LowerRewriter : StmtExprMutator {
+ public:
+  BF16LowerRewriter() {}
+
+  std::unordered_map<const BufferNode*, Buffer> buffer_remap;
+  std::unordered_map<const VarNode*, Var> var_remap;
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  PrimExpr VisitExpr_(const CastNode* op) final {
+    auto op_val = StmtExprMutator::VisitExpr(op->value);
+    if (op->value->dtype.is_bfloat16()) {
+      // if is cast_from_bf16, check if is to fp32
+      CHECK(op->dtype.is_float() && op->dtype.bits() == 32);
+      auto uint32_dtype = DataType(kDLUInt, 32, op_val->dtype.lanes());
+      auto uint32_v = CastNode::make(uint32_dtype, op_val);
+      return CallNode::make(op->dtype, CallNode::reinterpret, {uint32_v << 16},

Review comment:
       add a note for future update in endian invariant like 
   
   https://github.com/apache/incubator-tvm/blob/95b3ad971dfed6aa4e9f18d4f43c846806fd5292/src/runtime/stackvm/stackvm.cc#L211

##########
File path: src/tir/transforms/bf16_legalize.cc
##########
@@ -0,0 +1,384 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file bf16_legalize.cc
+ * \brief legalize bf16 type by adding cast_to_fp32
+ */
+
+#include <tvm/runtime/registry.h>
+#include <tvm/tir/op.h>
+#include <tvm/tir/transform.h>
+
+#include <cmath>
+#include <tuple>
+
+#include "../../arith/ir_mutator_with_analyzer.h"
+#include "../../arith/ir_visitor_with_analyzer.h"
+
+namespace tvm {
+namespace tir {
+
+using arith::Analyzer;
+using arith::IRMutatorWithAnalyzer;
+
+class BF16PromoteRewriter : public StmtExprMutator {
+ public:
+  BF16PromoteRewriter() {}
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  std::tuple<PrimExpr, PrimExpr> DoCast(PrimExpr orig_a, PrimExpr orig_b, bool* is_bfloat16) {
+    auto a = this->VisitExpr(orig_a);
+    auto b = this->VisitExpr(orig_b);
+    *is_bfloat16 = false;
+    if (a->dtype.is_bfloat16()) {
+      CHECK(b->dtype.is_bfloat16());
+      *is_bfloat16 = true;
+    } else if (b->dtype.is_bfloat16()) {
+      CHECK(a->dtype.is_bfloat16());
+      *is_bfloat16 = true;
+    }
+
+    if (*is_bfloat16) {
+      DataType fp32ty(kDLFloat, 32, 1);
+      a = CastNode::make(fp32ty, a);
+      b = CastNode::make(fp32ty, b);
+    }
+    return std::make_tuple(a, b);
+  }
+
+  PrimExpr VisitExpr_(const AddNode* op) final;
+  PrimExpr VisitExpr_(const SubNode* op) final;
+  PrimExpr VisitExpr_(const MulNode* op) final;
+  PrimExpr VisitExpr_(const DivNode* op) final;
+  PrimExpr VisitExpr_(const MinNode* op) final;
+  PrimExpr VisitExpr_(const MaxNode* op) final;
+  PrimExpr VisitExpr_(const LTNode* op) final;
+  PrimExpr VisitExpr_(const LENode* op) final;
+  PrimExpr VisitExpr_(const GTNode* op) final;
+  PrimExpr VisitExpr_(const GENode* op) final;
+};
+
+#define DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(OP, FUNC)        \
+  PrimExpr BF16PromoteRewriter::VisitExpr_(const OP* op) {       \
+    PrimExpr a, b;                                               \
+    bool is_bfloat16;                                            \
+    std::tie(a, b) = DoCast(op->a, op->b, &is_bfloat16);         \
+    if (a.same_as(op->a) && b.same_as(op->b)) {                  \
+      return GetRef<PrimExpr>(op);                               \
+    } else {                                                     \
+      auto ret = FUNC(a, b);                                     \
+      if (!is_bfloat16)                                          \
+        return ret;                                              \
+      else                                                       \
+        return CastNode::make(DataType(kTVMBFloat, 16, 1), ret); \
+    }                                                            \
+  }
+
+#define DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(OP, FUNC) \
+  PrimExpr BF16PromoteRewriter::VisitExpr_(const OP* op) {        \
+    PrimExpr a, b;                                                \
+    bool is_bfloat16;                                             \
+    std::tie(a, b) = DoCast(op->a, op->b, &is_bfloat16);          \
+    if (a.same_as(op->a) && b.same_as(op->b)) {                   \
+      return GetRef<PrimExpr>(op);                                \
+    } else {                                                      \
+      auto ret = FUNC(a, b);                                      \
+      return ret;                                                 \
+    }                                                             \
+  }
+
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(AddNode, operator+)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(SubNode, operator-)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MulNode, operator*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(DivNode, div)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MinNode, min)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MaxNode, max)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(LTNode, operator<)   // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(LENode, operator<=)  // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(GTNode, operator>)   // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(GENode, operator>=)  // NOLINT(*)
+
+/*
+ * Eliminate verbose casting between fp32 and bf16
+ * Checks if the AST has the pattern:
+ *     castto32(castto16(some_fp32_op(...)))
+ * The verbose casting is generated by BF16Promote for multiple
+ * bf16 Ops in a row. e.g.:
+ *  X[i] + Y[i] + T[i] =>
+ *  bf16((float32(bf16((float32(X[i]) + float32(Y[i])))) + float32(T[i])))
+ * After this pass:
+ *  bf16(float32(X[i]) + float32(Y[i]) + float32(T[i]))
+ */
+class BF16CastEliminationRewriter : public StmtExprMutator {
+ public:
+  BF16CastEliminationRewriter() {}
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  PrimExpr VisitExpr_(const CastNode* op) final {
+    auto op_val = StmtExprMutator::VisitExpr(op->value);
+    if (op->dtype.is_float() && op->dtype.bits() == 32) {
+      // if is cast_to_fp32, check if op->value is cast_to_fp16
+      // and op->value->value is a float32
+      if (auto innercast = op_val.as<CastNode>()) {
+        if (innercast->dtype.is_bfloat16() && innercast->value->dtype.is_float() &&
+            innercast->value->dtype.bits() == 32) {
+          return innercast->value;
+        }
+      }
+    }
+    if (op->value.same_as(op_val)) return GetRef<PrimExpr>(op);
+    return CastNode::make(op->dtype, op_val);
+  }
+};
+
+// implementation from
+// https://github.com/pytorch/pytorch/blob/master/c10/util/BFloat16.h
+inline uint16_t round_to_nearest_even(float src) {
+  if (std::isnan(src)) {
+    return UINT16_C(0x7FC0);
+  } else {
+    union {
+      uint32_t U32;
+      float F32;
+    };
+
+    F32 = src;
+    uint32_t rounding_bias = ((U32 >> 16) & 1) + UINT32_C(0x7FFF);
+    return static_cast<uint16_t>((U32 + rounding_bias) >> 16);
+  }
+}
+
+/*
+ * Lower the bf16 type to int16
+ * Lower cast between bf16 and fp32
+ * Lower bf16 FloatImm to int16
+ */
+class BF16LowerRewriter : StmtExprMutator {
+ public:
+  BF16LowerRewriter() {}
+
+  std::unordered_map<const BufferNode*, Buffer> buffer_remap;
+  std::unordered_map<const VarNode*, Var> var_remap;
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  PrimExpr VisitExpr_(const CastNode* op) final {
+    auto op_val = StmtExprMutator::VisitExpr(op->value);
+    if (op->value->dtype.is_bfloat16()) {
+      // if is cast_from_bf16, check if is to fp32
+      CHECK(op->dtype.is_float() && op->dtype.bits() == 32);
+      auto uint32_dtype = DataType(kDLUInt, 32, op_val->dtype.lanes());
+      auto uint32_v = CastNode::make(uint32_dtype, op_val);
+      return CallNode::make(op->dtype, CallNode::reinterpret, {uint32_v << 16},
+                            CallNode::PureIntrinsic);
+
+    } else if (op->dtype.is_bfloat16()) {
+      // if is cast_to_bf16, check if op->value is fp32
+      CHECK(op->value->dtype.is_float() && op->value->dtype.bits() == 32);
+      auto uint32_dtype = DataType(kDLUInt, 32, op_val->dtype.lanes());
+      auto uint32_v =
+          CallNode::make(uint32_dtype, CallNode::reinterpret, {op_val}, CallNode::PureIntrinsic);
+      auto uint16_dtype = DataType(kDLUInt, 16, op_val->dtype.lanes());
+      // uint32_t rounding_bias = ((U32 >> 16) & 1) + UINT32_C(0x7FFF);

Review comment:
       clean up

##########
File path: src/tir/transforms/bf16_legalize.cc
##########
@@ -0,0 +1,384 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file bf16_legalize.cc
+ * \brief legalize bf16 type by adding cast_to_fp32
+ */
+
+#include <tvm/runtime/registry.h>
+#include <tvm/tir/op.h>
+#include <tvm/tir/transform.h>
+
+#include <cmath>
+#include <tuple>
+
+#include "../../arith/ir_mutator_with_analyzer.h"
+#include "../../arith/ir_visitor_with_analyzer.h"
+
+namespace tvm {
+namespace tir {
+
+using arith::Analyzer;
+using arith::IRMutatorWithAnalyzer;
+
+class BF16PromoteRewriter : public StmtExprMutator {
+ public:
+  BF16PromoteRewriter() {}
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  std::tuple<PrimExpr, PrimExpr> DoCast(PrimExpr orig_a, PrimExpr orig_b, bool* is_bfloat16) {
+    auto a = this->VisitExpr(orig_a);
+    auto b = this->VisitExpr(orig_b);
+    *is_bfloat16 = false;
+    if (a->dtype.is_bfloat16()) {
+      CHECK(b->dtype.is_bfloat16());
+      *is_bfloat16 = true;
+    } else if (b->dtype.is_bfloat16()) {
+      CHECK(a->dtype.is_bfloat16());
+      *is_bfloat16 = true;
+    }
+
+    if (*is_bfloat16) {
+      DataType fp32ty(kDLFloat, 32, 1);
+      a = CastNode::make(fp32ty, a);
+      b = CastNode::make(fp32ty, b);
+    }
+    return std::make_tuple(a, b);
+  }
+
+  PrimExpr VisitExpr_(const AddNode* op) final;
+  PrimExpr VisitExpr_(const SubNode* op) final;
+  PrimExpr VisitExpr_(const MulNode* op) final;
+  PrimExpr VisitExpr_(const DivNode* op) final;
+  PrimExpr VisitExpr_(const MinNode* op) final;
+  PrimExpr VisitExpr_(const MaxNode* op) final;
+  PrimExpr VisitExpr_(const LTNode* op) final;
+  PrimExpr VisitExpr_(const LENode* op) final;
+  PrimExpr VisitExpr_(const GTNode* op) final;
+  PrimExpr VisitExpr_(const GENode* op) final;
+};
+
+#define DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(OP, FUNC)        \
+  PrimExpr BF16PromoteRewriter::VisitExpr_(const OP* op) {       \
+    PrimExpr a, b;                                               \
+    bool is_bfloat16;                                            \
+    std::tie(a, b) = DoCast(op->a, op->b, &is_bfloat16);         \
+    if (a.same_as(op->a) && b.same_as(op->b)) {                  \
+      return GetRef<PrimExpr>(op);                               \
+    } else {                                                     \
+      auto ret = FUNC(a, b);                                     \
+      if (!is_bfloat16)                                          \
+        return ret;                                              \
+      else                                                       \
+        return CastNode::make(DataType(kTVMBFloat, 16, 1), ret); \
+    }                                                            \
+  }
+
+#define DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(OP, FUNC) \
+  PrimExpr BF16PromoteRewriter::VisitExpr_(const OP* op) {        \
+    PrimExpr a, b;                                                \
+    bool is_bfloat16;                                             \
+    std::tie(a, b) = DoCast(op->a, op->b, &is_bfloat16);          \
+    if (a.same_as(op->a) && b.same_as(op->b)) {                   \
+      return GetRef<PrimExpr>(op);                                \
+    } else {                                                      \
+      auto ret = FUNC(a, b);                                      \
+      return ret;                                                 \
+    }                                                             \
+  }
+
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(AddNode, operator+)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(SubNode, operator-)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MulNode, operator*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(DivNode, div)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MinNode, min)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH(MaxNode, max)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(LTNode, operator<)   // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(LENode, operator<=)  // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(GTNode, operator>)   // NOLINT(*)
+DEFINE_BIOP_EXPR_MUTATE_WITH_TYPE_MATCH_NO_CAST(GENode, operator>=)  // NOLINT(*)
+
+/*
+ * Eliminate verbose casting between fp32 and bf16
+ * Checks if the AST has the pattern:
+ *     castto32(castto16(some_fp32_op(...)))
+ * The verbose casting is generated by BF16Promote for multiple
+ * bf16 Ops in a row. e.g.:
+ *  X[i] + Y[i] + T[i] =>
+ *  bf16((float32(bf16((float32(X[i]) + float32(Y[i])))) + float32(T[i])))
+ * After this pass:
+ *  bf16(float32(X[i]) + float32(Y[i]) + float32(T[i]))
+ */
+class BF16CastEliminationRewriter : public StmtExprMutator {
+ public:
+  BF16CastEliminationRewriter() {}
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  PrimExpr VisitExpr_(const CastNode* op) final {
+    auto op_val = StmtExprMutator::VisitExpr(op->value);
+    if (op->dtype.is_float() && op->dtype.bits() == 32) {
+      // if is cast_to_fp32, check if op->value is cast_to_fp16
+      // and op->value->value is a float32
+      if (auto innercast = op_val.as<CastNode>()) {
+        if (innercast->dtype.is_bfloat16() && innercast->value->dtype.is_float() &&
+            innercast->value->dtype.bits() == 32) {
+          return innercast->value;
+        }
+      }
+    }
+    if (op->value.same_as(op_val)) return GetRef<PrimExpr>(op);
+    return CastNode::make(op->dtype, op_val);
+  }
+};
+
+// implementation from
+// https://github.com/pytorch/pytorch/blob/master/c10/util/BFloat16.h
+inline uint16_t round_to_nearest_even(float src) {
+  if (std::isnan(src)) {
+    return UINT16_C(0x7FC0);
+  } else {
+    union {
+      uint32_t U32;
+      float F32;
+    };
+
+    F32 = src;
+    uint32_t rounding_bias = ((U32 >> 16) & 1) + UINT32_C(0x7FFF);
+    return static_cast<uint16_t>((U32 + rounding_bias) >> 16);
+  }
+}
+
+/*
+ * Lower the bf16 type to int16
+ * Lower cast between bf16 and fp32
+ * Lower bf16 FloatImm to int16
+ */
+class BF16LowerRewriter : StmtExprMutator {
+ public:
+  BF16LowerRewriter() {}
+
+  std::unordered_map<const BufferNode*, Buffer> buffer_remap;
+  std::unordered_map<const VarNode*, Var> var_remap;
+
+  Stmt operator()(Stmt s) { return VisitStmt(s); }
+
+  PrimExpr VisitExpr_(const CastNode* op) final {
+    auto op_val = StmtExprMutator::VisitExpr(op->value);
+    if (op->value->dtype.is_bfloat16()) {
+      // if is cast_from_bf16, check if is to fp32
+      CHECK(op->dtype.is_float() && op->dtype.bits() == 32);
+      auto uint32_dtype = DataType(kDLUInt, 32, op_val->dtype.lanes());
+      auto uint32_v = CastNode::make(uint32_dtype, op_val);
+      return CallNode::make(op->dtype, CallNode::reinterpret, {uint32_v << 16},
+                            CallNode::PureIntrinsic);
+
+    } else if (op->dtype.is_bfloat16()) {
+      // if is cast_to_bf16, check if op->value is fp32
+      CHECK(op->value->dtype.is_float() && op->value->dtype.bits() == 32);
+      auto uint32_dtype = DataType(kDLUInt, 32, op_val->dtype.lanes());
+      auto uint32_v =
+          CallNode::make(uint32_dtype, CallNode::reinterpret, {op_val}, CallNode::PureIntrinsic);
+      auto uint16_dtype = DataType(kDLUInt, 16, op_val->dtype.lanes());
+      // uint32_t rounding_bias = ((U32 >> 16) & 1) + UINT32_C(0x7FFF);
+      // return static_cast<uint16_t>((U32 + rounding_bias) >> 16);
+      auto rounding_bias = ((uint32_v >> 16) & 1) + make_const(uint16_dtype, 0x7FFF);
+      return CastNode::make(uint16_dtype, {(uint32_v + rounding_bias) >> 16});

Review comment:
       add a note for future update in endian invariant like 
   
   https://github.com/apache/incubator-tvm/blob/95b3ad971dfed6aa4e9f18d4f43c846806fd5292/src/runtime/stackvm/stackvm.cc#L211




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r434640906



##########
File path: python/tvm/_ffi/runtime_ctypes.py
##########
@@ -58,7 +58,8 @@ class DataType(ctypes.Structure):
         0 : 'int',
         1 : 'uint',
         2 : 'float',
-        4 : 'handle'
+        4 : 'handle',
+        65: 'bfloat'

Review comment:
       The definition of DataType code and PackedFunc argument type code starts to diverge after 3




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#issuecomment-635945266


   RFC Discussion link
   https://discuss.tvm.ai/t/rfc-add-bfloat16-data-type/6778


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#issuecomment-639344395


   @tqchen Thanks for the clarification. I have changed kTVMNullPtr back to 4.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#issuecomment-639143314


   https://github.com/apache/incubator-tvm/pull/5730 splits the two type codes, we only need to add BFLoat16 to the DataTypeCode. cc @Menooker 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] tqchen commented on pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#issuecomment-645017563


   @vinx13 @ZihengJiang @liangfu it would be great if you cam take another look and https://tvm.apache.org/docs/contribute/code_review.html#approve-and-request-changes-explicitly


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker commented on a change in pull request #5601: [DataType] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker commented on a change in pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#discussion_r441250264



##########
File path: python/tvm/_ffi/_cython/base.pxi
##########
@@ -27,7 +27,7 @@ cdef enum TVMTypeCode:
     kUInt = 1
     kFloat = 2
     kTVMOpaqueHandle = 3
-    kTVMNullptr = 4
+    kBFloat = 4

Review comment:
       changed




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Menooker edited a comment on pull request #5601: [TVM Core] Add bfloat16

Posted by GitBox <gi...@apache.org>.
Menooker edited a comment on pull request #5601:
URL: https://github.com/apache/incubator-tvm/pull/5601#issuecomment-633214226


   @tqchen I have changed the legalization pass to pure TIR->TIR. Added a pass called `BF16Lowering` to replace bf16 type with uint16 and lower the casting by shifting.
   
   I have met a problem that I need to replace `BufferNode` of bf16 with uint16. It works well and can produce same result of numpy. However, a lowered bf16 cast cannot pass `assert_structural_equal` test, like [here](https://github.com/Menooker/incubator-tvm/blob/177f99a8ce7f95e927cabeaf18684ade0c8ebc73/tests/python/unittest/test_tir_transform_bf16_legalize.py#L143). Is this a problem of this PR? I have checked the string output, the lowered TIR strings are the same. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org