[GitHub] [incubator-mxnet] Zha0q1 commented on issue #18926: ILP64 OpenBLAS Behaviors Different from Regular 32-bit LP64 OpenBLAS

[GitBox <gi...@apache.org>]

Zha0q1 commented on issue #18926:
URL: https://github.com/apache/incubator-mxnet/issues/18926#issuecomment-674333272


   > Thanks Zhaoqi. This is applicable for tensors with greater than 2pow31 right? It doesn't have any implications for regular tensors?
   
   After more discussion with openblas people my understanding is that loss of precision will happen whenever two numbers to be added are certain magnitude apart from each other. (in fact loss of precision always happen but is especially bad under such condition)
   
   Direct quotes from them:
   
   > Expected precision is as of standard netlib blas with single addition per loop. If you need to sum vaues more than significand+1 apart you need bigger floating point variable with obviously bigger significand range.
   
   > What the naive loop reaches is limit of "significand"
   > https://en.wikipedia.org/wiki/Floating-point_arithmetic#IEEE_754:_floating_point_in_modern_computers
   > i.e 2^24 (it rounds properly even near the end) i.e close to 1,6777216 ^ 10^7
   
   With that said much less error will be accumulated with regularly sized tensors. Also if all the entries to be added are more or less of of the same magnitude there will also be little loss of precision. Regular openblas is able to avoid this error accumulation by FMA/blocking so hence the discrepancy we saw between ilp64 openblas and vanilla openblas.
   
   With that said, my demo program calculates (1, 1, 1, ....) dot (1, 1, 1, ...).T. The error will start to accumulate when the sum reaches significand.
   
   With shape = (2^31, 1)
   ```
   2147483647 // 2^31
   1.93274e+09 // gemm answer
   2.14748e+09 // accurate value
   ```
   With shape = (2^29)
   ```
   536870911 // 2^29
   5.36871e+08 // ilp64 openblas gemm answer
   5.36871e+08 // accurate value
   ```
   
   compared with vanilla openblas
   With shape = (2^31, 1)
   ```
   2147483647 // 2^31
   2.14748e+09 // gemm answer
   2.14748e+09 // accurate value
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org