[GitHub] [incubator-tvm] AllenBSL opened a new issue #4446: [RFC] Fixed-point type implementation proposal

[GitBox <gi...@apache.org>]

AllenBSL opened a new issue #4446: [RFC] Fixed-point type implementation proposal
URL: https://github.com/apache/incubator-tvm/issues/4446
 
 
   While we were trying to implement RISC-V p extension in TVM, we found the arithmetic instructions in RISC-V 
   p extension only worked with integer type. RISC-V p extension supports fixed point instructions. Therefore, we tried to add fixed-point type in TVM to quantize the inference programs from floating-point type to fixed-point type.
   
   ## Transform Floating-point to Fixed-point
   
   The value of a fixed-point type variable is determined by the value of its floating-point counter part and the point position which can be set by the users. The higher point position means more accuracy in decimal part after quantization. However, there will be less bits to express integer part thus resulting a smaller number range. In another word, we could set a higher point position if the data set consists with smaller number range. Otherwise, the quantiztion will cause too much saturation and result in low accuracy prediction from inference program. 
   
   The actual relationship between fixed-point variable and floating-point variable can be depicted as follow:
   
   We refer Fxp as fixed-point value, Fp as floating-point value and PP as point position
   
   $Fxp = Fp * pow(2,PP)$
   
   We can see that point position is actually the exponent of 2, we multiply it with floating-point value and retain the integer part as fixed-point value
   
   e.g.
   If we set point position as 10. The floating-point value 0.25 is equal to 256 in fixed-point value. (0.25*pow(2,10) = 256)
   
   ## Arithmetic of Fixed-point
   
   The arithmetic operations of fixed-point are similar to the ones of integer, but there are some operations need some tweaks to get the correct result.
   
   e.g.
   One of the most common operation we will encounter is multiplication. Assume point position is 10. The fixed-point value of 0.5 is 512. If we multiply 0.5 with itself, we should get 0.25(256 in fixed-point) as result. However, we won't get same result from multiplication of fixed-point value. We have to divide it with pow(2,10) for the correct answer(or shift right 10 digits).
   
   In order to get the correct arithmetic result in fixed-point, the binary point position is an essential information.
   
   
   ## Implementing Fixed-point type in TVM
   
   If we want to build a NN model with fixed-point type while using nnvm or relay as below, we need to set the fixed-point type first in python code.
   
   ```
   func, params = relay.frontend.from_mxnet(block, shape_dict, "fxp16_13")
   with relay.build_config(opt_level=level):
               graph, lib, params = relay.build(func, target, params=params)
   ```
   
   In this case, the fixed-point type is presented as 'fxp16_13'. 16 is the number of bits in the fixed-point variable. 13 is the point position.
   
   We use llvm as our target in experiment. The main goal is to pass down the information of point position to backend compiler. The method is to add a new type in TVM.
   
   ### TVM IR Type
   
   We base our fixed-point type on HalideIR::Type. It is similar to HalideIR::Int. The only different is that we use 'halideir_handle_cplusplus_type' to carry the point position.
   
   ```
   inline Type Fxp(int bits, int fxp_pos, int lanes = 1) {
       halideir_handle_cplusplus_type *fxp_info = new halideir_handle_cplusplus_type{
               halideir_cplusplus_type_name(halideir_cplusplus_type_name::Simple, "fxp16_"+std::to_string(fxp_pos)),
               {}, {}, {}};
       return Type(Type::Int,bits, lanes,fxp_info);
   }
   
   ```
   
   ### TVMType
   
   Since TVMType is costructed by three integer variables (code, bits and lanes). There is no variable to carry point position. We have to give each possible point position a unique type code to distinguish them from each other. If we want to add fxp16 type into TVMType, we will need 16 codes(numbers) for each possible position.
   
   ### Codegen
   
   As mentioned above, we need to add extra instructions so fixed-point multiplication would return correct result. Our method is to modify the codegen of llvm and override 'VisitExpr_(const Mul* op)' function to verify the fixed-point multiplication. The type class in codegen phase is TVM IR type and we can get the data of point position from it. Then we add shift-right instruction according to the point position and generate the fixed-point version of multiplication in llvm IR form.
   
   In our experiment, we use a llvm intrinsic to replace normal integer multiplications with fixed-point multiplications. 
   ```
   llvm::Value* CodeGenRISCV::VisitExpr_(const Mul* op) {
   
       int fxp_pos=0;
           
       if(op->type.handle_type!=nullptr) {
         std::string fxp_name = op->type.handle_type->inner_name.name;
         if(fxp_name.find("fxp")!=std::string::npos)
           fxp_pos = std::stoi(fxp_name.substr(fxp_name.find("_")+1));
       }
       
       if(op->type.bits()==16||op->type.bits()==8) {
         int total_bits = op->type.bits()*op->type.lanes();
         if(total_bits==32||total_bits==64) {
           Expr e = CreateMulIntr(op,op->type.bits(),op->type.lanes(),fxp_pos);
           return CodeGenCPU::CreateIntrinsic(e.as<Call>());
         }       
       }    
       return CreateMul(op->type, MakeValue(op->a), MakeValue(op->b));  
       
   }
   ```
   
   
   We should use llvm backend compiler to compile llvm IR and get the correct multiplication library with fixed-point type. 
   
   Thanks for reading and welcome to comment
   
   -Bing-Sung Lu (Allen Lu), Chao-Lin Lee from Peakhills Group AI team,
     Jenq-Kuen Lee, YI-RU CHEN from NTHU

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services