You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by "yongwww (via GitHub)" <gi...@apache.org> on 2024/06/17 03:02:29 UTC

[I] [Bug] correctness issue of scaled_dot_product_attention [tvm]

yongwww opened a new issue, #17099:
URL: https://github.com/apache/tvm/issues/17099

   Currently, the Torch function `F.scaled_dot_product_attention` is mapped to `R.nn.attention` in both the Relax nn.module and the FX converter. However, the inference results do not seem to match those obtained with PyTorch. Script to trigger the issue as below.
   
   ```
   import tvm
   from tvm import relax
   from tvm.script.parser import ir as I, relax as R
   import numpy as np
   import torch
   import torch.nn.functional as F
   
   # TVM: commit cc7eb2faae3444ee02b142a5aea237dd1db6d29a
   # Torch: 2.3.1
   np.random.seed(11111)
   shape = (2, 24, 4250, 64)
   dtype = "float32"
   
   qkv_pt = []
   qkv_tvm = []
   for _ in range(3):
       val = np.random.normal(loc=0, scale=1, size=shape).astype(dtype)
       qkv_pt.append(torch.tensor(val).to("cuda"))
       qkv_tvm.append(tvm.nd.array(val, tvm.cuda()))
   
   
   @I.ir_module
   class Model:
       @R.function
       def main(
           query: R.Tensor(shape, dtype=dtype),
           key: R.Tensor(shape, dtype=dtype),
           value: R.Tensor(shape, dtype=dtype),
       ) -> R.Tensor(shape, dtype=dtype):
           R.func_attr({"num_input": 3})
           with R.dataflow():
               gv: R.Tensor(shape, dtype=dtype) = R.nn.attention(query, key, value)
               R.output(gv)
           return gv
   
   target = tvm.target.Target("cuda") # "nvidia/nvidia-a100"
   ex = relax.build(Model, target)
   vm = relax.VirtualMachine(ex, tvm.cuda())
   tvm_output = vm["main"](*qkv_tvm)
   
   torch_out = F.scaled_dot_product_attention(*qkv_pt)
   print(f"tvm output: {tvm_output}\ntorch output: {torch_out}")
   np.testing.assert_allclose(tvm_output.numpy(), torch_out.cpu().numpy(), atol=1e-5)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] correctness issue of scaled_dot_product_attention [tvm]

Posted by "yongwww (via GitHub)" <gi...@apache.org>.
yongwww closed issue #17099: [Bug] correctness issue of scaled_dot_product_attention
URL: https://github.com/apache/tvm/issues/17099


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] correctness issue of scaled_dot_product_attention [tvm]

Posted by "yongwww (via GitHub)" <gi...@apache.org>.
yongwww commented on issue #17099:
URL: https://github.com/apache/tvm/issues/17099#issuecomment-2172148511

   extra transpose works.
   
   ```
           with R.dataflow():
               q = R.permute_dims(query, [0, 2, 1, 3])
               k = R.permute_dims(key, [0, 2, 1, 3])
               v = R.permute_dims(value, [0, 2, 1, 3])
               r = R.nn.attention(q, k, v)
               gv = R.permute_dims(r, [0, 2, 1, 3])
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org