You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by "mshr-h (via GitHub)" <gi...@apache.org> on 2024/09/16 14:23:54 UTC
[PR] Fix torch sdpa converter [tvm]
mshr-h opened a new pull request, #17379:
URL: https://github.com/apache/tvm/pull/17379
[torch.nn.functional.scaled_dot_product_attention](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) outputs in the shape of `(N, ..., L, E_v)` but `relax.op.nn.attention` does `(N, L, ..., E_v)` so the output should also be transposed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Relax][PyTorch] Fix output shape of `torch.nn.functional.scaled_dot_product_attention` [tvm]
Posted by "yongwww (via GitHub)" <gi...@apache.org>.
yongwww commented on PR #17379:
URL: https://github.com/apache/tvm/pull/17379#issuecomment-2354467192
we can transpose to get the expected result. Thanks for the effort!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Relax][PyTorch] Fix output shape of `torch.nn.functional.scaled_dot_product_attention` [tvm]
Posted by "mshr-h (via GitHub)" <gi...@apache.org>.
mshr-h commented on code in PR #17379:
URL: https://github.com/apache/tvm/pull/17379#discussion_r1762274939
##########
tests/python/nightly/relax/test_frontend_from_fx.py:
##########
@@ -0,0 +1,70 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import torch
+from torch import fx
+from torch.nn import Module
+
+import tvm
+from tvm import relax
+import tvm.testing
+from tvm.relax.frontend.torch import from_fx
Review Comment:
Ok. Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Relax][PyTorch] Fix output shape of `torch.nn.functional.scaled_dot_product_attention` [tvm]
Posted by "mshr-h (via GitHub)" <gi...@apache.org>.
mshr-h commented on PR #17379:
URL: https://github.com/apache/tvm/pull/17379#issuecomment-2355276758
MSC E2E test is failing. Seems like we also need to change something other than relax frontend.
@Archermmt Do you have any ideas on how to fix the error?
https://ci.tlcpack.ai/blue/organizations/jenkins/tvm-unity/detail/PR-17379/6/pipeline/
> [2024-09-17T10:06:53.212Z] tests/python/contrib/test_msc/test_translate_torch.py::test_attention FAILED
> [2024-09-17T10:06:53.212Z]
> [2024-09-17T10:06:53.212Z] =================================== FAILURES ===================================
> [2024-09-17T10:06:53.212Z] ________________________________ test_attention ________________________________
> [2024-09-17T10:06:53.212Z]
> [2024-09-17T10:06:53.212Z] def test_attention():
> [2024-09-17T10:06:53.212Z] """test torch translator for attention"""
> [2024-09-17T10:06:53.212Z]
> [2024-09-17T10:06:53.212Z] # pylint: disable=import-outside-toplevel
> [2024-09-17T10:06:53.212Z] import torch.nn.functional as F
> [2024-09-17T10:06:53.212Z]
> [2024-09-17T10:06:53.212Z] class Attention1(Module):
> [2024-09-17T10:06:53.212Z] def forward(self, q_data, k_data, v_data):
> [2024-09-17T10:06:53.212Z] return F.scaled_dot_product_attention(q_data, k_data, v_data)
> [2024-09-17T10:06:53.212Z]
> [2024-09-17T10:06:53.212Z] class Attention2(Module):
> [2024-09-17T10:06:53.212Z] def forward(self, q_data, k_data, v_data):
> [2024-09-17T10:06:53.212Z] return F.scaled_dot_product_attention(q_data, k_data, v_data, is_causal=True)
> [2024-09-17T10:06:53.212Z]
> [2024-09-17T10:06:53.212Z] input_info = [
> [2024-09-17T10:06:53.212Z] ([32, 8, 128, 64], "float32"),
> [2024-09-17T10:06:53.212Z] ([32, 8, 128, 64], "float32"),
> [2024-09-17T10:06:53.212Z] ([32, 8, 128, 64], "float32"),
> [2024-09-17T10:06:53.212Z] ]
> [2024-09-17T10:06:53.212Z] > verify_model(Attention1(), input_info)
> [2024-09-17T10:06:53.212Z]
> [2024-09-17T10:06:53.212Z] tests/python/contrib/test_msc/test_translate_torch.py:1127:
> [2024-09-17T10:06:53.212Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> [2024-09-17T10:06:53.212Z] tests/python/contrib/test_msc/test_translate_torch.py:52: in verify_model
> [2024-09-17T10:06:53.212Z] tvm.testing.assert_allclose(
> [2024-09-17T10:06:53.212Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> [2024-09-17T10:06:53.212Z]
> [2024-09-17T10:06:53.212Z] actual = array([[[[0.5253085 , 0.48211107, 0.52921075, ..., 0.518867 ,
> [2024-09-17T10:06:53.212Z] 0.49926636, 0.48493868],
> [2024-09-17T10:06:53.212Z] [0.5294311 ...5],
> [2024-09-17T10:06:53.212Z] [0.47335747, 0.48579183, 0.5360674 , ..., 0.543607 ,
> [2024-09-17T10:06:53.212Z] 0.5020893 , 0.47848547]]]], dtype=float32)
> [2024-09-17T10:06:53.212Z] desired = array([[[[0.5253085 , 0.48211107, 0.52921075, ..., 0.518867 ,
> [2024-09-17T10:06:53.212Z] 0.49926636, 0.48493868],
> [2024-09-17T10:06:53.212Z] [0.49697113... ],
> [2024-09-17T10:06:53.212Z] [0.47335747, 0.48579183, 0.5360674 , ..., 0.543607 ,
> [2024-09-17T10:06:53.212Z] 0.5020893 , 0.47848547]]]], dtype=float32)
> [2024-09-17T10:06:53.212Z] rtol = 1e-05, atol = 1e-05
> [2024-09-17T10:06:53.212Z]
> [2024-09-17T10:06:53.212Z] def assert_allclose(actual, desired, rtol=1e-7, atol=1e-7):
> [2024-09-17T10:06:53.212Z] """Version of np.testing.assert_allclose with `atol` and `rtol` fields set
> [2024-09-17T10:06:53.212Z] in reasonable defaults.
> [2024-09-17T10:06:53.212Z]
> [2024-09-17T10:06:53.212Z] Arguments `actual` and `desired` are not interchangeable, since the function
> [2024-09-17T10:06:53.212Z] compares the `abs(actual-desired)` with `atol+rtol*abs(desired)`. Since we
> [2024-09-17T10:06:53.212Z] often allow `desired` to be close to zero, we generally want non-zero `atol`.
> [2024-09-17T10:06:53.212Z] """
> [2024-09-17T10:06:53.212Z] actual = np.asanyarray(actual)
> [2024-09-17T10:06:53.212Z] desired = np.asanyarray(desired)
> [2024-09-17T10:06:53.212Z] > np.testing.assert_allclose(actual.shape, desired.shape)
> [2024-09-17T10:06:53.212Z] E AssertionError:
> [2024-09-17T10:06:53.212Z] E Not equal to tolerance rtol=1e-07, atol=0
> [2024-09-17T10:06:53.212Z] E
> [2024-09-17T10:06:53.212Z] E Mismatched elements: 2 / 4 (50%)
> [2024-09-17T10:06:53.212Z] E Max absolute difference: 120
> [2024-09-17T10:06:53.212Z] E Max relative difference: 15.
> [2024-09-17T10:06:53.212Z] E x: array([ 32, 8, 128, 64])
> [2024-09-17T10:06:53.212Z] E y: array([ 32, 128, 8, 64])
> [2024-09-17T10:06:53.212Z]
> [2024-09-17T10:06:53.212Z] python/tvm/testing/utils.py:119: AssertionError
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Relax][PyTorch] Fix output shape of `torch.nn.functional.scaled_dot_product_attention` [tvm]
Posted by "yongwww (via GitHub)" <gi...@apache.org>.
yongwww commented on code in PR #17379:
URL: https://github.com/apache/tvm/pull/17379#discussion_r1762263503
##########
tests/python/nightly/relax/test_frontend_from_fx.py:
##########
@@ -0,0 +1,70 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import torch
+from torch import fx
+from torch.nn import Module
+
+import tvm
+from tvm import relax
+import tvm.testing
+from tvm.relax.frontend.torch import from_fx
Review Comment:
I think we might not need this nightly test, as the test case changes in tests/python/relax seem sufficient to cover this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Relax][PyTorch] Fix output shape of `torch.nn.functional.scaled_dot_product_attention` [tvm]
Posted by "yongwww (via GitHub)" <gi...@apache.org>.
yongwww merged PR #17379:
URL: https://github.com/apache/tvm/pull/17379
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org