You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2022/03/15 20:47:23 UTC
[GitHub] [tvm] masahi opened a new pull request #10622: [Testing] Add model loader for int8 BERT
masahi opened a new pull request #10622:
URL: https://github.com/apache/tvm/pull/10622
Following https://github.com/tlc-pack/TLCBench/pull/5, add an easy API to load the int8 BERT model. Currently it sits under `meta_schedule/testing/tlcbench.py` but `relay/testing/tlcbench.py` is also fine.
@junrushao1994 @sunggg @AndrewZhaoLuo @mbrookhart @jwfromm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao1994 commented on a change in pull request #10622: [Testing] Add model loader for int8 BERT
Posted by GitBox <gi...@apache.org>.
junrushao1994 commented on a change in pull request #10622:
URL: https://github.com/apache/tvm/pull/10622#discussion_r827463197
##########
File path: python/tvm/meta_schedule/testing/tlcbench.py
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,import-outside-toplevel
+# type: ignore
+"""Model loader for TLCBench."""
+import os
+import logging
+import tvm
+from tvm import relay
+from tvm.error import TVMError
+from tvm.contrib.download import download_testdata
+
+
+log = logging.getLogger("tlcbench")
+
+
+def convert_to_qnn(onnx_path, json_path, params_path, batch_size, seq_len):
+ """Run the ONNX frontend and the FQ2I pass. The output is serialized to disk."""
+ import onnx
+
+ onnx_model = onnx.load(onnx_path)
+
+ shape_dict = {
+ "input_ids": (batch_size, seq_len),
+ "segment_ids": (batch_size, seq_len),
+ "input_mask": (batch_size, seq_len),
+ }
+
+ log.info("Converting te ONNX model to Relay and running the FQ2I pass, it may take a while...")
+ mod, params = relay.frontend.from_onnx(onnx_model, shape_dict, freeze_params=True)
+
+ seq = tvm.transform.Sequential(
+ [relay.transform.InferType(), relay.transform.FakeQuantizationToInteger(use_qat=True)]
+ )
+ mod = seq(mod)
+
+ with open(json_path, "w") as fo:
+ fo.write(tvm.ir.save_json(mod))
+
+ with open(params_path, "wb") as fo:
+ fo.write(relay.save_param_dict(params))
+
+
+def deserialize_relay(json_path, params_path):
+ with open(json_path, "r") as fi:
+ mod = tvm.ir.load_json(fi.read())
+
+ with open(params_path, "rb") as fi:
+ params = relay.load_param_dict(fi.read())
+
+ return mod, params
+
+
+def load_quantized_bert_base(batch_size=1, seq_len=384):
Review comment:
Looks like this importing process could be a bit complicated, so shall we do that in a separate process like what we did in `relay_workload.py` when importing from pytorch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #10622: [Testing] Add model loader for int8 BERT
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #10622:
URL: https://github.com/apache/tvm/pull/10622#discussion_r827528323
##########
File path: python/tvm/meta_schedule/testing/tlcbench.py
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,import-outside-toplevel
+# type: ignore
+"""Model loader for TLCBench."""
+import os
+import logging
+import tvm
+from tvm import relay
+from tvm.error import TVMError
+from tvm.contrib.download import download_testdata
+
+
+log = logging.getLogger("tlcbench")
+
+
+def convert_to_qnn(onnx_path, json_path, params_path, batch_size, seq_len):
+ """Run the ONNX frontend and the FQ2I pass. The output is serialized to disk."""
+ import onnx
+
+ onnx_model = onnx.load(onnx_path)
+
+ shape_dict = {
+ "input_ids": (batch_size, seq_len),
+ "segment_ids": (batch_size, seq_len),
+ "input_mask": (batch_size, seq_len),
+ }
+
+ log.info("Converting te ONNX model to Relay and running the FQ2I pass, it may take a while...")
+ mod, params = relay.frontend.from_onnx(onnx_model, shape_dict, freeze_params=True)
+
+ seq = tvm.transform.Sequential(
+ [relay.transform.InferType(), relay.transform.FakeQuantizationToInteger(use_qat=True)]
+ )
+ mod = seq(mod)
+
+ with open(json_path, "w") as fo:
+ fo.write(tvm.ir.save_json(mod))
+
+ with open(params_path, "wb") as fo:
+ fo.write(relay.save_param_dict(params))
+
+
+def deserialize_relay(json_path, params_path):
+ with open(json_path, "r") as fi:
+ mod = tvm.ir.load_json(fi.read())
+
+ with open(params_path, "rb") as fi:
+ params = relay.load_param_dict(fi.read())
+
+ return mod, params
+
+
+def load_quantized_bert_base(batch_size=1, seq_len=384):
Review comment:
Done in https://github.com/apache/tvm/pull/10622/commits/7bcbbad5f1346adb637fe8c367d347d4750020fc. Does this look ok? I don't understand why we want to do thing in a separate process tho, I don't see any difference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #10622: [Testing] Add model loader for int8 BERT
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #10622:
URL: https://github.com/apache/tvm/pull/10622#discussion_r827555204
##########
File path: python/tvm/meta_schedule/testing/tlcbench.py
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,import-outside-toplevel
+# type: ignore
+"""Model loader for TLCBench."""
+import os
+import logging
+import tvm
+from tvm import relay
+from tvm.error import TVMError
+from tvm.contrib.download import download_testdata
+
+
+log = logging.getLogger("tlcbench")
+
+
+def convert_to_qnn(onnx_path, json_path, params_path, batch_size, seq_len):
+ """Run the ONNX frontend and the FQ2I pass. The output is serialized to disk."""
+ import onnx
+
+ onnx_model = onnx.load(onnx_path)
+
+ shape_dict = {
Review comment:
but yeah, the current interface doesn't make it obvious what the input names are, which are needed for deployment. I can add a comment describing what the input names are.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] sunggg commented on a change in pull request #10622: [Testing] Add model loader for int8 BERT
Posted by GitBox <gi...@apache.org>.
sunggg commented on a change in pull request #10622:
URL: https://github.com/apache/tvm/pull/10622#discussion_r827556982
##########
File path: python/tvm/meta_schedule/testing/tlcbench.py
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,import-outside-toplevel
+# type: ignore
+"""Model loader for TLCBench."""
+import os
+import logging
+import tvm
+from tvm import relay
+from tvm.error import TVMError
+from tvm.contrib.download import download_testdata
+
+
+log = logging.getLogger("tlcbench")
+
+
+def convert_to_qnn(onnx_path, json_path, params_path, batch_size, seq_len):
+ """Run the ONNX frontend and the FQ2I pass. The output is serialized to disk."""
+ import onnx
+
+ onnx_model = onnx.load(onnx_path)
+
+ shape_dict = {
Review comment:
Oh really? I thought I needed to match that all the time.
I see. If that's the case, it looks good to me.
Maybe it would be better to fix `relay_workload.py` side for the consistency.
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #10622: [Testing] Add model loader for int8 BERT
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #10622:
URL: https://github.com/apache/tvm/pull/10622#discussion_r827562812
##########
File path: python/tvm/meta_schedule/testing/tlcbench.py
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,import-outside-toplevel
+# type: ignore
+"""Model loader for TLCBench."""
+import os
+import logging
+import tvm
+from tvm import relay
+from tvm.error import TVMError
+from tvm.contrib.download import download_testdata
+
+
+log = logging.getLogger("tlcbench")
+
+
+def convert_to_qnn(onnx_path, json_path, params_path, batch_size, seq_len):
+ """Run the ONNX frontend and the FQ2I pass. The output is serialized to disk."""
+ import onnx
+
+ onnx_model = onnx.load(onnx_path)
+
+ shape_dict = {
Review comment:
Sure, check out this commit https://github.com/apache/tvm/pull/10622/commits/752b812bdd83172cf9e9c26171aaacf7cba92c30
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #10622: [Testing] Add model loader for int8 BERT
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #10622:
URL: https://github.com/apache/tvm/pull/10622#discussion_r827558781
##########
File path: python/tvm/meta_schedule/testing/tlcbench.py
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,import-outside-toplevel
+# type: ignore
+"""Model loader for TLCBench."""
+import os
+import logging
+import tvm
+from tvm import relay
+from tvm.error import TVMError
+from tvm.contrib.download import download_testdata
+
+
+log = logging.getLogger("tlcbench")
+
+
+def convert_to_qnn(onnx_path, json_path, params_path, batch_size, seq_len):
+ """Run the ONNX frontend and the FQ2I pass. The output is serialized to disk."""
+ import onnx
+
+ onnx_model = onnx.load(onnx_path)
+
+ shape_dict = {
Review comment:
I mean, the names of inputs you pass to the PT frontend and the ones you use in `runtime.set_input(input_name, ...)` must match, off course. But for the PT the actual value of `input_name` can be whatever, while for onnx we need to use exactly the same name recorded in the input onnx model.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi merged pull request #10622: [Testing] Add model loader for int8 BERT
Posted by GitBox <gi...@apache.org>.
masahi merged pull request #10622:
URL: https://github.com/apache/tvm/pull/10622
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #10622: [Testing] Add model loader for int8 BERT
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #10622:
URL: https://github.com/apache/tvm/pull/10622#discussion_r827558781
##########
File path: python/tvm/meta_schedule/testing/tlcbench.py
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,import-outside-toplevel
+# type: ignore
+"""Model loader for TLCBench."""
+import os
+import logging
+import tvm
+from tvm import relay
+from tvm.error import TVMError
+from tvm.contrib.download import download_testdata
+
+
+log = logging.getLogger("tlcbench")
+
+
+def convert_to_qnn(onnx_path, json_path, params_path, batch_size, seq_len):
+ """Run the ONNX frontend and the FQ2I pass. The output is serialized to disk."""
+ import onnx
+
+ onnx_model = onnx.load(onnx_path)
+
+ shape_dict = {
Review comment:
I mean, the names of inputs you pass to the PT frontend and the ones you use in `runtime.set_input(input_name, ...`)` must match, off course. But for the PT the actual value of `input_name` can be whatever, while for onnx we need to use exactly the same name recorded in the input onnx model.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] sunggg commented on a change in pull request #10622: [Testing] Add model loader for int8 BERT
Posted by GitBox <gi...@apache.org>.
sunggg commented on a change in pull request #10622:
URL: https://github.com/apache/tvm/pull/10622#discussion_r827560344
##########
File path: python/tvm/meta_schedule/testing/tlcbench.py
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,import-outside-toplevel
+# type: ignore
+"""Model loader for TLCBench."""
+import os
+import logging
+import tvm
+from tvm import relay
+from tvm.error import TVMError
+from tvm.contrib.download import download_testdata
+
+
+log = logging.getLogger("tlcbench")
+
+
+def convert_to_qnn(onnx_path, json_path, params_path, batch_size, seq_len):
+ """Run the ONNX frontend and the FQ2I pass. The output is serialized to disk."""
+ import onnx
+
+ onnx_model = onnx.load(onnx_path)
+
+ shape_dict = {
Review comment:
Oh somehow your later comments were not shown when I was leaving the comment.
Yeah, personally, I would prefer returning `shape_dict` to the caller cause I'm a lazy person :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao1994 commented on a change in pull request #10622: [Testing] Add model loader for int8 BERT
Posted by GitBox <gi...@apache.org>.
junrushao1994 commented on a change in pull request #10622:
URL: https://github.com/apache/tvm/pull/10622#discussion_r827439045
##########
File path: python/tvm/meta_schedule/testing/tlcbench.py
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,import-outside-toplevel
+# type: ignore
+"""Model loader for TLCBench."""
+import os
+import logging
+import tvm
+from tvm import relay
+from tvm.error import TVMError
+from tvm.contrib.download import download_testdata
+
+
+log = logging.getLogger("tlcbench")
Review comment:
I'm not an expert in python logging, but looks like people prefer using `__name__`?
```suggestion
log = logging.getLogger(__name__)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao1994 commented on a change in pull request #10622: [Testing] Add model loader for int8 BERT
Posted by GitBox <gi...@apache.org>.
junrushao1994 commented on a change in pull request #10622:
URL: https://github.com/apache/tvm/pull/10622#discussion_r827535641
##########
File path: python/tvm/meta_schedule/testing/tlcbench.py
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,import-outside-toplevel
+# type: ignore
+"""Model loader for TLCBench."""
+import os
+import logging
+import tvm
+from tvm import relay
+from tvm.error import TVMError
+from tvm.contrib.download import download_testdata
+
+
+log = logging.getLogger("tlcbench")
+
+
+def convert_to_qnn(onnx_path, json_path, params_path, batch_size, seq_len):
+ """Run the ONNX frontend and the FQ2I pass. The output is serialized to disk."""
+ import onnx
+
+ onnx_model = onnx.load(onnx_path)
+
+ shape_dict = {
+ "input_ids": (batch_size, seq_len),
+ "segment_ids": (batch_size, seq_len),
+ "input_mask": (batch_size, seq_len),
+ }
+
+ log.info("Converting te ONNX model to Relay and running the FQ2I pass, it may take a while...")
+ mod, params = relay.frontend.from_onnx(onnx_model, shape_dict, freeze_params=True)
+
+ seq = tvm.transform.Sequential(
+ [relay.transform.InferType(), relay.transform.FakeQuantizationToInteger(use_qat=True)]
+ )
+ mod = seq(mod)
+
+ with open(json_path, "w") as fo:
+ fo.write(tvm.ir.save_json(mod))
+
+ with open(params_path, "wb") as fo:
+ fo.write(relay.save_param_dict(params))
+
+
+def deserialize_relay(json_path, params_path):
+ with open(json_path, "r") as fi:
+ mod = tvm.ir.load_json(fi.read())
+
+ with open(params_path, "rb") as fi:
+ params = relay.load_param_dict(fi.read())
+
+ return mod, params
+
+
+def load_quantized_bert_base(batch_size=1, seq_len=384):
Review comment:
looks good!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] sunggg commented on a change in pull request #10622: [Testing] Add model loader for int8 BERT
Posted by GitBox <gi...@apache.org>.
sunggg commented on a change in pull request #10622:
URL: https://github.com/apache/tvm/pull/10622#discussion_r827470337
##########
File path: python/tvm/meta_schedule/testing/tlcbench.py
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,import-outside-toplevel
+# type: ignore
+"""Model loader for TLCBench."""
+import os
+import logging
+import tvm
+from tvm import relay
+from tvm.error import TVMError
+from tvm.contrib.download import download_testdata
+
+
+log = logging.getLogger("tlcbench")
+
+
+def convert_to_qnn(onnx_path, json_path, params_path, batch_size, seq_len):
+ """Run the ONNX frontend and the FQ2I pass. The output is serialized to disk."""
+ import onnx
+
+ onnx_model = onnx.load(onnx_path)
+
+ shape_dict = {
Review comment:
Can we pass this info like we do in `relay_workload.py`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #10622: [Testing] Add model loader for int8 BERT
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #10622:
URL: https://github.com/apache/tvm/pull/10622#discussion_r827555627
##########
File path: python/tvm/meta_schedule/testing/tlcbench.py
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,import-outside-toplevel
+# type: ignore
+"""Model loader for TLCBench."""
+import os
+import logging
+import tvm
+from tvm import relay
+from tvm.error import TVMError
+from tvm.contrib.download import download_testdata
+
+
+log = logging.getLogger("tlcbench")
+
+
+def convert_to_qnn(onnx_path, json_path, params_path, batch_size, seq_len):
+ """Run the ONNX frontend and the FQ2I pass. The output is serialized to disk."""
+ import onnx
+
+ onnx_model = onnx.load(onnx_path)
+
+ shape_dict = {
Review comment:
or we can return `shape_dict` to the caller, that way the caller doesn't need to hard-code input names / shapes. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] sunggg commented on a change in pull request #10622: [Testing] Add model loader for int8 BERT
Posted by GitBox <gi...@apache.org>.
sunggg commented on a change in pull request #10622:
URL: https://github.com/apache/tvm/pull/10622#discussion_r827565699
##########
File path: python/tvm/meta_schedule/testing/tlcbench.py
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,import-outside-toplevel
+# type: ignore
+"""Model loader for TLCBench."""
+import os
+import logging
+import tvm
+from tvm import relay
+from tvm.error import TVMError
+from tvm.contrib.download import download_testdata
+
+
+log = logging.getLogger("tlcbench")
+
+
+def convert_to_qnn(onnx_path, json_path, params_path, batch_size, seq_len):
+ """Run the ONNX frontend and the FQ2I pass. The output is serialized to disk."""
+ import onnx
+
+ onnx_model = onnx.load(onnx_path)
+
+ shape_dict = {
Review comment:
Cool. LGTM. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #10622: [Testing] Add model loader for int8 BERT
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #10622:
URL: https://github.com/apache/tvm/pull/10622#discussion_r827554029
##########
File path: python/tvm/meta_schedule/testing/tlcbench.py
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,import-outside-toplevel
+# type: ignore
+"""Model loader for TLCBench."""
+import os
+import logging
+import tvm
+from tvm import relay
+from tvm.error import TVMError
+from tvm.contrib.download import download_testdata
+
+
+log = logging.getLogger("tlcbench")
+
+
+def convert_to_qnn(onnx_path, json_path, params_path, batch_size, seq_len):
+ """Run the ONNX frontend and the FQ2I pass. The output is serialized to disk."""
+ import onnx
+
+ onnx_model = onnx.load(onnx_path)
+
+ shape_dict = {
Review comment:
I don't think so, in pytorch case the names of inputs don't matter, but for onnx they need to match the one in the *.onnx file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #10622: [Testing] Add model loader for int8 BERT
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #10622:
URL: https://github.com/apache/tvm/pull/10622#discussion_r827562812
##########
File path: python/tvm/meta_schedule/testing/tlcbench.py
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,import-outside-toplevel
+# type: ignore
+"""Model loader for TLCBench."""
+import os
+import logging
+import tvm
+from tvm import relay
+from tvm.error import TVMError
+from tvm.contrib.download import download_testdata
+
+
+log = logging.getLogger("tlcbench")
+
+
+def convert_to_qnn(onnx_path, json_path, params_path, batch_size, seq_len):
+ """Run the ONNX frontend and the FQ2I pass. The output is serialized to disk."""
+ import onnx
+
+ onnx_model = onnx.load(onnx_path)
+
+ shape_dict = {
Review comment:
Sure, check out this commit https://github.com/apache/tvm/pull/10622/commits/752b812bdd83172cf9e9c26171aaacf7cba92c30 This way the caller doesn't need to care what the actual input names are.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org