[tvm-rfcs] branch main updated: [RFC][OpenCLML] OpenCLML integration as BYOC (#52)

[ma...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

masahi pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tvm-rfcs.git


The following commit(s) were added to refs/heads/main by this push:
     new f5ef65f  [RFC][OpenCLML] OpenCLML integration as BYOC (#52)
f5ef65f is described below

commit f5ef65fd0f04178af266ed109206ba3f011d8eaa
Author: Siva <qu...@quicinc.com>
AuthorDate: Sat Jan 29 10:17:46 2022 +0530

    [RFC][OpenCLML] OpenCLML integration as BYOC (#52)
    
    * [RFC][OpenCLML] OpenCLML integration as BYOC
    
    * [RFC][OpenCLML] Review comments.
---
 rfcs/0052-OpenCLML-integratio-as-BYOC.md | 90 ++++++++++++++++++++++++++++++++
 1 file changed, 90 insertions(+)

diff --git a/rfcs/0052-OpenCLML-integratio-as-BYOC.md b/rfcs/0052-OpenCLML-integratio-as-BYOC.md
new file mode 100644
index 0000000..304fa9a
--- /dev/null
+++ b/rfcs/0052-OpenCLML-integratio-as-BYOC.md
@@ -0,0 +1,90 @@
+- Feature Name: OpenCL ML integration as BYOC
+- Start Date: 2022-01-13
+- RFC PR: [apache/tvm-rfcs#52](https://github.com/apache/tvm-rfcs/pull/52)
+- GitHub Issue: TBD
+
+
+# Summary
+[summary]: #summary
+
+OpenCL ML is an extension (cl_qcom_ml_ops) over OpenCL spec developed by Qualcomm to accelerate the machine learning at operation level. OpenCL SDK is publicly available at OpenCL Machine Learning Acceleration on Adreno GPU - Qualcomm Developer Network. OpenCL ML leverages deep knowledge of Adreno GPU for significant performance benefits. It offers C based DNN API with compatibility to most of the standard frameworks. Its standard OpenCL features like command queues, buffers, events and  [...]
+
+# Motivation
+[motivation]: #motivation
+
+The current OpenCL backend of TVM is very generic and not optimized well for Adreno performance capabilities. Adreno GPU has quite a few proprietary and standard OpenCL paths. OpenCL ML extension offers accelerated ML operations via an SDK interface.
+
+With TVM having the entire framework of frontends, graph level optimizations and OpenCL ML having kernels that perform best on Adreno GPU, in this work we aim to integrate OpenCLML SDK into TVM as a BYOC. This effort brings best of both worlds where TVM handling high level optimizations, sub graphs are scheduled on OpenCL ML based on the support and the operators not supported by OpenCL ML will take TVM’s default OpenCL path. Good thing here is we don’t need separate OpenCL workspaces or [...]
+
+# Guide-level explanation
+[guide-level-explanation]: #guide-level-explanation
+
+This RFC aims to introduce OpenCLML runtime as a BYOC option into TVM. In terms of usage, it’s very similar to other BYOC integrations we have in TVM.
+
+Along with all other options we use for OpenCL target, here we introduce the below build options in config.cmake
+
+```USE_CLML``` (ON/OFF) This enables CLML codegen for compilation
+
+```USE_CLML_GRAPH_EXECUTOR``` (ON/OFF) This enables CLML runtime
+Btw, OpenCLML SDK provides replacement for default libOpenCL.so. Hence, we don’t need a separate option to point OpenCLML SDK instead just point OpenCLML SDK path for USE_OPENCL.
+
+Introduces front end helper API as ```tvm.relay.op.contrib.clml```. This will help to partition the graph and annotating the subgraphs to OpenCL CLML target.
+
+Given mod and params that represents TVM Module and params the below API does partitioning based on OpenCLML support.
+
+```mod = clml.partition_for_clml(mod, params)```
+
+Post above partitioning we just follow standard ```relay.build``` process.
+
+Talking about runtime, OpenCL ML runtime compilation is same as OpenCL compilation for Android target. Just that USE_OPENCL points to OpenCL ML SDK.
+
+# Reference-level explanation
+[reference-level-explanation]: #reference-level-explanation
+
+
+Like any other BYOC implementation this RFC enhances/introduces a frontend helper API for partitioning, a codegen for CLML and CLML runtime.
+
+### Frontend:
+Front end implements ```tvm.relay.op.contrib.clml``` user API ```partition_for_clml``` and ```is_clml_runtime_enabled``` for partitioning the relay graph to OpenCLML path. It also contains clml specific patten table definition and other transform helpers required for CLML target.
+
+### Codegen:
+CLML codegen built over JSONSerializer. Thanks to JSONSerializer for all the infra here and one can focus only on target specific parsing and JSON Node generation. The codegen exports ```relay.ext.clml```, ```relay.op.is_clml_runtime_enabled``` into TVM global space.
+
+### Runtime:
+OpenCLML Runtime is again extended over JSONRuntimeBase and implements OpenCL ML initialization, implementation of CLML API invocation corresponding to CLML annotated layers.
+
+OpenCLML runtime support is verified by looking for ```cl_qcom_ml_ops``` into OpenCL extension list.
+
+OpenCLML doesn’t define a new open context instead it reused the context defined by OpenCL runtime through global API ```device_api.opencl```.
+
+OpenCLML has its own CLML tensor objects called ```cl_ml_tensor_memory_desc_qcom```. The runtime defines the copy API from OpenCL to CLML Tensors within the same OpenCL work space without bringing the data back to host.
+
+OpenCLML supports tuning too which generally produces a tuning cache file and reuses for later runs. This implementation supports looking for environment variable ```CLML_IS_TUNNING_RUN``` set to 0/1 to run for tuning and also supports ```CLML_TUNNING_CACHE``` to set the tuning cache file location. While implementation CLML tuning happens at last step of ```BuildEngine``` by calling ```clTuneMLOpQCOM``` followed by ```clSaveMLTuningCacheQCOM``` for saving the cache to given file. Later w [...]
+
+# Drawbacks
+[drawbacks]: #drawbacks
+
+
+OpenCLML is supported by Snapdragon devices only with extension ```cl_qcom_ml_ops```. Seamless copy from OpenCL to CLML is supported for clBuffers now. Using Image objects on TVM may have challenges for direct copy within OpenCL context.
+
+# Rationale and alternatives
+[rationale-and-alternatives]: #rationale-and-alternatives
+
+
+OpenCL ML uses Adreno specific proprietary and public optimization paths and outperforms TVM generated OpenCL kernels by a big difference.
+
+# Prior art
+[prior-art]: #prior-art
+
+There exists an ongoing development for texture support on Adreno devices https://discuss.tvm.apache.org/t/rfc-texture-memory-support/9467.
+
+# Unresolved questions
+[unresolved-questions]: #unresolved-questions
+
+
+How do we deal with sub graphs with tiny layers? This is the case where not offloading the tiny layer performs better than accelerator.
+
+# Future possibilities
+[future-possibilities]: #future-possibilities
+
+Integrating OpenCLML into TVM gives an end-to-end compiler stack for Snapdragon platform with Adreno GPU target. Operator support evolves along with OpenCL ML SDK releases from Qualcomm.