You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by tq...@apache.org on 2026/04/29 18:48:36 UTC
(tvm) branch main updated: [REFACTOR][RUNTIME][CODEGEN] Backend specific target and runtime to enable cross-compile fallback (#19465)
This is an automated email from the ASF dual-hosted git repository.
tqchen pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tvm.git
The following commit(s) were added to refs/heads/main by this push:
new 6e8f77d664 [REFACTOR][RUNTIME][CODEGEN] Backend specific target and runtime to enable cross-compile fallback (#19465)
6e8f77d664 is described below
commit 6e8f77d664e3a0c4424aaa3d753c170503d03796
Author: Tianqi Chen <tq...@users.noreply.github.com>
AuthorDate: Wed Apr 29 14:48:31 2026 -0400
[REFACTOR][RUNTIME][CODEGEN] Backend specific target and runtime to enable cross-compile fallback (#19465)
## Why
This refactor reshapes each backend into a self-contained
`src/target/<X>/`
cluster (with optional `src/target/<X>/llvm/` for LLVM-dependent
codegen) and
introduces a per-backend fallback module that absorbs the cross-compile
role
cleanly — without `target/opt/` stubs, without `DeviceSourceModuleNode`,
and
without leaking a synthetic `kind()` to consumers.
## High-level principles
- **One directory per backend.** All codegen-side files for backend
`<X>` live
under `src/target/<X>/`. Optional `src/target/<X>/llvm/` subdir for
files
that require `USE_LLVM` at build time. Backend grouping wins over
build-dependency grouping (the latter being upstream's `target/llvm/`).
- **Plugin-only runtime modules.** `src/runtime/<X>/<X>_module.h` is
deleted.
The runtime's real `<X>ModuleNode` is reachable only via the FFI
registry
(`ffi.Module.create.<kind>`, `ffi.Module.load_from_bytes.<kind>`). No
C++ API surface other than the static registrations.
- **Per-backend fallback module for cross-compile.** Each `<X>` gets a
`<X>FallbackModuleNode` in `src/target/<X>/<X>_fallback_module.{h,cc}`.
Same `kind()` as the real module. Codegen-time only — never reachable
via
load. `GetFunction` errors with a backend-specific "runtime not linked"
message; `InspectSource` works.
- **Codegen-side wrapper does the fallback selection.** Codegen calls
`<X>ModuleCreateWithFallback(...)`, which tries
`ffi.Module.create.<kind>`
via the registry; on miss, falls through to `<X>FallbackModuleCreate`
(plain C++; reachable directly from the fallback header). When
`USE_<X>=ON`
is in effect, the registry hit returns the real module; when
`USE_<X>=OFF`, the fallback is what codegen gets. No CMake `if/else`
gating; fallback always compiled.
## Specific changes
### New per-backend directories (codegen + fallback)
- `src/target/cuda/` — `codegen_cuda.cc` + `intrin_rule_cuda.cc` +
fallback module pair + `llvm/codegen_nvptx.cc`
- `src/target/rocm/` — fallback module pair + `llvm/codegen_amdgpu.cc` +
`llvm/intrin_rule_rocm.cc`
- `src/target/hexagon/` — fallback module pair +
`llvm/codegen_hexagon.cc` + `llvm/intrin_rule_hexagon.cc`
- `src/target/metal/` — `codegen_metal.cc` + `intrin_rule_metal.cc` +
fallback module pair
- `src/target/vulkan/` — `build_vulkan.cc` + the rest of `target/spirv/`
absorbed + fallback module pair
- `src/target/opencl/` — `codegen_opencl.cc` + `intrin_rule_opencl.cc` +
fallback module pair
- `src/target/webgpu/` — `codegen_webgpu.cc` + fallback module pair (was
`WebGPUSourceModuleNode`, renamed)
### New fallback classes
`<X>FallbackModuleNode` for X in {`CUDA`, `ROCm`, `Hexagon`, `Metal`,
`Vulkan`, `OpenCL`, `WebGPU`}. Each:
- `kind()` matches the real backend
- Stores `(code or smap, fmt, fmap, source)` — no driver/runtime calls
- `GetFunction` errors with backend-specific "runtime not linked"
message
- `InspectSource` works fully
- `SaveToBytes` byte-identical to real
---
CMakeLists.txt | 7 +
cmake/modules/CUDA.cmake | 3 -
cmake/modules/Hexagon.cmake | 10 +-
cmake/modules/LLVM.cmake | 7 +-
cmake/modules/Metal.cmake | 6 +-
cmake/modules/OpenCL.cmake | 3 -
cmake/modules/ROCM.cmake | 2 -
cmake/modules/Vulkan.cmake | 13 +-
include/tvm/support/serializer.h | 21 +++
python/tvm/contrib/nvcc.py | 12 +-
.../tvm/script/ir_builder/tirx/external_kernel.py | 26 +--
src/runtime/cuda/cuda_module.cc | 178 +++++++++++++--------
src/runtime/cuda/cuda_module.h | 66 --------
src/runtime/file_utils.cc | 23 ++-
src/runtime/hexagon/hexagon_module.cc | 163 ++++++++++++-------
src/runtime/hexagon/hexagon_module.h | 95 -----------
src/runtime/metal/metal_module.h | 61 -------
src/runtime/metal/metal_module.mm | 153 +++++++++---------
src/runtime/opencl/opencl_common.h | 27 ++--
src/runtime/opencl/opencl_module.cc | 107 +++++++------
src/runtime/opencl/opencl_module.h | 92 -----------
src/runtime/opencl/opencl_module_spirv.cc | 159 ------------------
src/runtime/rocm/rocm_module.cc | 112 +++++++------
src/runtime/rocm/rocm_module.h | 66 --------
src/runtime/spirv/spirv_shader.h | 30 ----
src/runtime/vulkan/vulkan_common.h | 3 -
src/runtime/vulkan/vulkan_module.cc | 76 ++++-----
src/runtime/vulkan/vulkan_module.h | 74 ---------
src/runtime/vulkan/vulkan_wrapped_func.cc | 36 ++---
src/runtime/vulkan/vulkan_wrapped_func.h | 32 ++--
src/target/{source => cuda}/codegen_cuda.cc | 57 +++++++
src/target/{source => cuda}/codegen_cuda.h | 8 +-
src/target/cuda/cuda_fallback_module.cc | 120 ++++++++++++++
src/target/cuda/cuda_fallback_module.h | 86 ++++++++++
src/target/{source => cuda}/intrin_rule_cuda.cc | 0
src/target/{source => cuda}/literal/cuda_half_t.h | 0
src/target/{source => cuda}/literal/cuda_int8_t.h | 0
src/target/{ => cuda}/llvm/codegen_nvptx.cc | 16 +-
src/target/{source => cuda}/ptx.cc | 0
src/target/{source => cuda}/ptx.h | 0
src/target/hexagon/hexagon_fallback_module.cc | 130 +++++++++++++++
src/target/hexagon/hexagon_fallback_module.h | 92 +++++++++++
src/target/{ => hexagon}/llvm/codegen_hexagon.cc | 27 +++-
.../{ => hexagon}/llvm/intrin_rule_hexagon.cc | 2 +-
src/target/{source => metal}/codegen_metal.cc | 17 +-
src/target/{source => metal}/codegen_metal.h | 8 +-
src/target/{source => metal}/intrin_rule_metal.cc | 0
src/target/metal/metal_fallback_module.cc | 131 +++++++++++++++
src/target/metal/metal_fallback_module.h | 89 +++++++++++
src/target/{source => opencl}/codegen_opencl.cc | 20 ++-
src/target/{source => opencl}/codegen_opencl.h | 8 +-
.../{source => opencl}/intrin_rule_opencl.cc | 0
src/target/opencl/opencl_fallback_module.cc | 128 +++++++++++++++
src/target/opencl/opencl_fallback_module.h | 87 ++++++++++
src/target/opt/build_cuda_off.cc | 24 ---
src/target/opt/build_cuda_on.cc | 112 -------------
src/target/opt/build_hexagon_off.cc | 49 ------
src/target/opt/build_metal_off.cc | 24 ---
src/target/opt/build_opencl_off.cc | 53 ------
src/target/opt/build_rocm_off.cc | 24 ---
src/target/{ => rocm}/llvm/codegen_amdgpu.cc | 15 +-
src/target/{ => rocm}/llvm/intrin_rule_rocm.cc | 4 +-
src/target/rocm/rocm_fallback_module.cc | 126 +++++++++++++++
src/target/rocm/rocm_fallback_module.h | 86 ++++++++++
src/target/{spirv => vulkan}/build_vulkan.cc | 26 ++-
src/target/{spirv => vulkan}/codegen_spirv.cc | 0
src/target/{spirv => vulkan}/codegen_spirv.h | 8 +-
src/target/{spirv => vulkan}/intrin_rule_spirv.cc | 0
src/target/{spirv => vulkan}/ir_builder.cc | 0
src/target/{spirv => vulkan}/ir_builder.h | 6 +-
src/target/{spirv => vulkan}/spirv_support.cc | 0
src/target/{spirv => vulkan}/spirv_support.h | 6 +-
src/target/{spirv => vulkan}/spirv_utils.cc | 2 +-
src/target/{spirv => vulkan}/spirv_utils.h | 8 +-
src/target/vulkan/vulkan_fallback_module.cc | 125 +++++++++++++++
src/target/vulkan/vulkan_fallback_module.h | 88 ++++++++++
src/target/{source => webgpu}/codegen_webgpu.cc | 71 ++------
src/target/{source => webgpu}/codegen_webgpu.h | 8 +-
.../{source => webgpu}/intrin_rule_webgpu.cc | 0
src/target/webgpu/webgpu_fallback_module.cc | 139 ++++++++++++++++
src/target/webgpu/webgpu_fallback_module.h | 90 +++++++++++
tests/python/codegen/test_target_codegen_cuda.py | 38 ++++-
tests/python/codegen/test_target_codegen_metal.py | 26 +++
tests/python/codegen/test_target_codegen_opencl.py | 38 +++++
tests/python/codegen/test_target_codegen_rocm.py | 39 +++++
tests/python/codegen/test_target_codegen_vulkan.py | 26 +++
86 files changed, 2471 insertions(+), 1479 deletions(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 8056a1d780..ee6726c1b9 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -296,6 +296,13 @@ tvm_file_glob(GLOB CODEGEN_SRCS
src/target/source/*.cc
src/target/canonicalizer/*.cc
src/target/canonicalizer/llvm/*.cc
+ src/target/cuda/*.cc
+ src/target/rocm/*.cc
+ src/target/hexagon/*.cc
+ src/target/metal/*.cc
+ src/target/opencl/*.cc
+ src/target/vulkan/vulkan_fallback_module.cc
+ src/target/webgpu/*.cc
)
list(APPEND COMPILER_SRCS ${CODEGEN_SRCS})
diff --git a/cmake/modules/CUDA.cmake b/cmake/modules/CUDA.cmake
index be0c7b3828..a79e558837 100644
--- a/cmake/modules/CUDA.cmake
+++ b/cmake/modules/CUDA.cmake
@@ -52,7 +52,6 @@ if(USE_CUDA)
tvm_file_glob(GLOB RUNTIME_CUDA_SRCS src/runtime/cuda/*.cc)
list(APPEND RUNTIME_SRCS ${RUNTIME_CUDA_SRCS})
- list(APPEND COMPILER_SRCS src/target/opt/build_cuda_on.cc)
list(APPEND TVM_RUNTIME_LINKER_LIBS ${CUDA_CUDART_LIBRARY})
list(APPEND TVM_RUNTIME_LINKER_LIBS ${CUDA_CUDA_LIBRARY})
@@ -137,6 +136,4 @@ if(USE_CUDA)
# Add CUDA builtins to RelaxVM
tvm_file_glob(GLOB VM_CUDA_BUILTIN_SRC_CC src/runtime/vm/cuda/*.cc)
list(APPEND RUNTIME_SRCS ${VM_CUDA_BUILTIN_SRC_CC})
-else(USE_CUDA)
- list(APPEND COMPILER_SRCS src/target/opt/build_cuda_off.cc)
endif(USE_CUDA)
diff --git a/cmake/modules/Hexagon.cmake b/cmake/modules/Hexagon.cmake
index 6db8f663ad..5995374408 100644
--- a/cmake/modules/Hexagon.cmake
+++ b/cmake/modules/Hexagon.cmake
@@ -78,12 +78,10 @@ endif()
if(NOT USE_HEXAGON)
- # If nothing related to Hexagon is enabled, add phony Hexagon codegen,
- # and some stuff needed by cpptests (this part is a temporary workaround
- # until e2e support for Hexagon is enabled).
- if(BUILD_FOR_HOST)
- list(APPEND COMPILER_SRCS src/target/opt/build_hexagon_off.cc)
- endif()
+ # USE_HEXAGON=OFF: codegen still works through the per-backend fallback
+ # module (src/target/hexagon/hexagon_fallback_module.cc), which is always
+ # compiled into libtvm via CODEGEN_SRCS. No opt-stub registration is
+ # needed.
return()
endif()
diff --git a/cmake/modules/LLVM.cmake b/cmake/modules/LLVM.cmake
index a204b29751..f944b41304 100644
--- a/cmake/modules/LLVM.cmake
+++ b/cmake/modules/LLVM.cmake
@@ -42,7 +42,12 @@ if(NOT ${USE_LLVM} MATCHES ${IS_FALSE_PATTERN})
add_definitions(-DTVM_MLIR_VERSION=${TVM_MLIR_VERSION})
endif()
add_definitions(-DTVM_LLVM_HAS_AARCH64_TARGET=${TVM_LLVM_HAS_AARCH64_TARGET})
- tvm_file_glob(GLOB COMPILER_LLVM_SRCS src/target/llvm/*.cc)
+ tvm_file_glob(GLOB COMPILER_LLVM_SRCS
+ src/target/llvm/*.cc
+ src/target/cuda/llvm/*.cc
+ src/target/rocm/llvm/*.cc
+ src/target/hexagon/llvm/*.cc
+ )
list(APPEND TVM_LINKER_LIBS ${LLVM_LIBS})
list(APPEND COMPILER_SRCS ${COMPILER_LLVM_SRCS})
if(NOT MSVC)
diff --git a/cmake/modules/Metal.cmake b/cmake/modules/Metal.cmake
index 78ccba4ba3..a9f0e9dd53 100644
--- a/cmake/modules/Metal.cmake
+++ b/cmake/modules/Metal.cmake
@@ -22,7 +22,7 @@ if(USE_METAL)
tvm_file_glob(GLOB RUNTIME_METAL_SRCS src/runtime/metal/*.mm)
list(APPEND TVM_RUNTIME_LINKER_LIBS ${METAL_LIB} ${FOUNDATION_LIB})
list(APPEND RUNTIME_SRCS ${RUNTIME_METAL_SRCS})
-
-else(USE_METAL)
- list(APPEND COMPILER_SRCS src/target/opt/build_metal_off.cc)
endif(USE_METAL)
+# When USE_METAL=OFF the codegen-side fallback in
+# src/target/metal/metal_fallback_module.cc handles construction; no opt
+# stub is needed (it is always compiled via CODEGEN_SRCS in CMakeLists.txt).
diff --git a/cmake/modules/OpenCL.cmake b/cmake/modules/OpenCL.cmake
index a01462d64c..3076c5f227 100644
--- a/cmake/modules/OpenCL.cmake
+++ b/cmake/modules/OpenCL.cmake
@@ -17,7 +17,6 @@
if(USE_OPENCL)
tvm_file_glob(GLOB RUNTIME_OPENCL_SRCS src/runtime/opencl/*.cc)
- list(APPEND COMPILER_SRCS src/target/spirv/spirv_utils.cc)
if(${USE_OPENCL} MATCHES ${IS_TRUE_PATTERN})
message(STATUS "Enabled runtime search for OpenCL library location")
@@ -50,6 +49,4 @@ if(USE_OPENCL)
message(STATUS "Set OpenCL Target version to " ${CMAKE_MATCH_1})
endif()
endif(USE_OPENCL_EXTN_QCOM)
-else()
- list(APPEND COMPILER_SRCS src/target/opt/build_opencl_off.cc)
endif(USE_OPENCL)
diff --git a/cmake/modules/ROCM.cmake b/cmake/modules/ROCM.cmake
index fe5b72c887..366dde3a69 100644
--- a/cmake/modules/ROCM.cmake
+++ b/cmake/modules/ROCM.cmake
@@ -67,6 +67,4 @@ if(USE_ROCM)
list(APPEND TVM_RUNTIME_LINKER_LIBS roc::rocthrust)
endif(USE_THRUST)
-else(USE_ROCM)
- list(APPEND COMPILER_SRCS src/target/opt/build_rocm_off.cc)
endif(USE_ROCM)
diff --git a/cmake/modules/Vulkan.cmake b/cmake/modules/Vulkan.cmake
index 1f303f3a03..bce4dada88 100644
--- a/cmake/modules/Vulkan.cmake
+++ b/cmake/modules/Vulkan.cmake
@@ -29,7 +29,18 @@ if(USE_VULKAN)
include_directories(SYSTEM ${Vulkan_INCLUDE_DIRS})
message(STATUS "Build with Vulkan support")
tvm_file_glob(GLOB RUNTIME_VULKAN_SRCS src/runtime/vulkan/*.cc)
- tvm_file_glob(GLOB COMPILER_VULKAN_SRCS src/target/spirv/*.cc)
+ # SPIR-V codegen tooling lives under src/target/vulkan/ alongside the
+ # fallback module. The fallback module itself is always compiled (in
+ # CMakeLists.txt's CODEGEN_SRCS); the rest depends on spirv-tools and
+ # is only compiled when USE_VULKAN=ON.
+ tvm_file_glob(GLOB COMPILER_VULKAN_SRCS
+ src/target/vulkan/build_vulkan.cc
+ src/target/vulkan/codegen_spirv.cc
+ src/target/vulkan/intrin_rule_spirv.cc
+ src/target/vulkan/ir_builder.cc
+ src/target/vulkan/spirv_support.cc
+ src/target/vulkan/spirv_utils.cc
+ )
list(APPEND RUNTIME_SRCS ${RUNTIME_VULKAN_SRCS})
list(APPEND COMPILER_SRCS ${COMPILER_VULKAN_SRCS})
list(APPEND TVM_LINKER_LIBS ${Vulkan_SPIRV_TOOLS_LIBRARY})
diff --git a/include/tvm/support/serializer.h b/include/tvm/support/serializer.h
index 8c1c136b1a..57852945f8 100644
--- a/include/tvm/support/serializer.h
+++ b/include/tvm/support/serializer.h
@@ -211,6 +211,27 @@ struct Serializer<ffi::String> {
}
};
+// ---- ffi::Bytes (binary payload, same wire format as std::string) ----
+template <>
+struct Serializer<ffi::Bytes> {
+ static constexpr bool enabled = true;
+
+ static void Write(Stream* strm, const ffi::Bytes& data) {
+ uint64_t sz = static_cast<uint64_t>(data.size());
+ Serializer<uint64_t>::Write(strm, sz);
+ if (sz != 0) {
+ strm->Write(data.data(), data.size());
+ }
+ }
+
+ static bool Read(Stream* strm, ffi::Bytes* data) {
+ std::string s;
+ if (!Serializer<std::string>::Read(strm, &s)) return false;
+ *data = ffi::Bytes(std::move(s));
+ return true;
+ }
+};
+
// ---- ffi::Array<T> (binary-compatible with std::vector<T>) ----
template <typename T>
struct Serializer<ffi::Array<T>> {
diff --git a/python/tvm/contrib/nvcc.py b/python/tvm/contrib/nvcc.py
index 1f933a63fc..33c12edd76 100644
--- a/python/tvm/contrib/nvcc.py
+++ b/python/tvm/contrib/nvcc.py
@@ -770,12 +770,15 @@ def find_nvshmem_paths() -> tuple[str, str]:
@tvm_ffi.register_global_func
-def tvm_callback_cuda_compile(code, target): # pylint: disable=unused-argument
+def tvm_callback_cuda_compile(code):
"""
Compile CUDA code using the configured backend (nvcc or nvrtc).
This callback is invoked by TVM's C++ backend during CUDA module compilation.
- By default, uses nvcc to generate fatbin.
+ By default, uses nvcc to generate fatbin. The current target is fetched
+ inside the callback (via ``tvm.target.Target.current(allow_none=True)``)
+ so the caller does not need to push/pop a target scope around the
+ invocation.
Environment Variables
---------------------
@@ -788,14 +791,15 @@ def tvm_callback_cuda_compile(code, target): # pylint: disable=unused-argument
----------
code : str
CUDA source code to compile
- target : Target
- TVM target architecture
Returns
-------
bytes
Compiled binary (fatbin for nvcc, cubin for nvrtc)
"""
+ # The current Target is fetched inside compile_cuda via
+ # tvm.target.Target.current(allow_none=True) when arch is unset; the
+ # caller no longer needs to push/pop a target scope.
compiler = os.environ.get("TVM_CUDA_COMPILE_MODE", "nvcc").lower()
if compiler == "nvrtc":
diff --git a/python/tvm/script/ir_builder/tirx/external_kernel.py b/python/tvm/script/ir_builder/tirx/external_kernel.py
index ebe432d69a..1785454a0f 100644
--- a/python/tvm/script/ir_builder/tirx/external_kernel.py
+++ b/python/tvm/script/ir_builder/tirx/external_kernel.py
@@ -24,10 +24,12 @@ import tempfile
from pathlib import Path
from typing import Any
+import tvm_ffi
+
from tvm import __version__ as tvm_version
from tvm import tirx
from tvm.contrib import nvcc
-from tvm.runtime import Module, const, load_module
+from tvm.runtime import Module, const
class BaseKernel: # pylint: disable=too-few-public-methods
@@ -91,17 +93,17 @@ class BaseKernel: # pylint: disable=too-few-public-methods
tvm_metadata = self._format_tvm_module_metadata(
kernel_name, kernel_arg_types, launch_param_tags
)
- with tempfile.TemporaryDirectory() as temp_dir:
- binary_path = f"{temp_dir}/{kernel_name}.{fmt}"
- if fmt == "ptx":
- with open(binary_path, "w") as f:
- f.write(binary_data)
- else:
- with open(binary_path, "wb") as f:
- f.write(binary_data)
- with open(f"{temp_dir}/{kernel_name}.tvm_meta.json", "w") as f:
- f.write(tvm_metadata)
- kernel_module = load_module(binary_path)
+ # Build the FunctionInfo map in-memory from the JSON metadata, then
+ # construct the CUDA module via the FFI registry without going to
+ # disk. Avoids the load_from_file dispatch path entirely.
+ if isinstance(binary_data, str):
+ binary_bytes = binary_data.encode("utf-8")
+ else:
+ binary_bytes = bytes(binary_data)
+ load_meta = tvm_ffi.get_global_func("runtime.LoadMetaDataFromJSON")
+ fmap = load_meta(tvm_metadata)
+ create_cuda = tvm_ffi.get_global_func("ffi.Module.create.cuda")
+ kernel_module = create_cuda(binary_bytes, fmt, fmap, {})
return kernel_module
diff --git a/src/runtime/cuda/cuda_module.cc b/src/runtime/cuda/cuda_module.cc
index 29a04eb185..1375ef8e76 100644
--- a/src/runtime/cuda/cuda_module.cc
+++ b/src/runtime/cuda/cuda_module.cc
@@ -19,15 +19,17 @@
/*!
* \file cuda_module.cc
+ * \brief CUDAModuleNode — runtime-side, plugin-only. Reachable from C++ only
+ * through the FFI registry keys "ffi.Module.create.cuda" and
+ * "ffi.Module.load_from_bytes.cuda". No exported header — codegen-side
+ * construction goes through src/target/cuda/cuda_fallback_module.h.
*/
-#include "cuda_module.h"
-
#include <cuda.h>
#include <cuda_runtime.h>
#include <tvm/ffi/extra/c_env_api.h>
+#include <tvm/ffi/extra/module.h>
#include <tvm/ffi/function.h>
#include <tvm/ffi/reflection/registry.h>
-#include <tvm/support/io.h>
#include <array>
#include <mutex>
@@ -35,7 +37,6 @@
#include <vector>
#include "../../support/bytes_io.h"
-#include "../file_utils.h"
#include "../metadata.h"
#include "../pack_args.h"
#include "../thread_storage_scope.h"
@@ -44,15 +45,18 @@
namespace tvm {
namespace runtime {
+// Maximum number of GPU supported in CUDAModule (file-local).
+static constexpr const int kMaxNumGPUs = 32;
+
// Module to support thread-safe multi-GPU execution.
// cuModule is a per-GPU module
// The runtime will contain a per-device module table
// The modules will be lazily loaded
class CUDAModuleNode : public ffi::ModuleObj {
public:
- explicit CUDAModuleNode(std::string data, std::string fmt,
- ffi::Map<ffi::String, FunctionInfo> fmap, std::string cuda_source)
- : data_(data), fmt_(fmt), fmap_(fmap), cuda_source_(cuda_source) {
+ CUDAModuleNode(ffi::Bytes code, ffi::String fmt, ffi::Map<ffi::String, FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source)
+ : code_(code), fmt_(fmt), fmap_(fmap), source_(source) {
std::fill(module_.begin(), module_.end(), nullptr);
}
// destructor
@@ -79,37 +83,42 @@ class CUDAModuleNode : public ffi::ModuleObj {
ffi::Optional<ffi::Function> GetFunction(const ffi::String& name) final;
- void WriteToFile(const ffi::String& file_name, const ffi::String& format) const final {
- std::string fmt = GetFileFormat(file_name, format);
- std::string meta_file = GetMetaFilePath(file_name);
- if (fmt == "cu") {
- TVM_FFI_ICHECK_NE(cuda_source_.length(), 0);
- SaveMetaDataToFile(meta_file, fmap_);
- SaveBinaryToFile(file_name, cuda_source_);
- } else {
- TVM_FFI_ICHECK_EQ(fmt, fmt_) << "Can only save to format=" << fmt_;
- SaveMetaDataToFile(meta_file, fmap_);
- SaveBinaryToFile(file_name, data_);
- }
- }
-
ffi::Bytes SaveToBytes() const final {
- std::string result;
- support::BytesOutStream stream(&result);
+ // Format: [fmt][fmap][code]. Source map is in-memory inspection only and
+ // is NEVER serialized — it is lost on save/load round-trip (matches
+ // upstream behavior; the receiver rebuilds source from code bytes if
+ // possible). CUDAFallbackModuleNode::SaveToBytes (in
+ // src/target/cuda/cuda_fallback_module.cc) MUST mirror this format
+ // byte-for-byte; see one-way comment there.
+ std::string buffer;
+ support::BytesOutStream stream(&buffer);
stream.Write(fmt_);
stream.Write(fmap_);
- stream.Write(data_);
- return ffi::Bytes(std::move(result));
+ stream.Write(code_);
+ return ffi::Bytes(std::move(buffer));
}
ffi::String InspectSource(const ffi::String& format) const final {
- if (format == fmt_) return data_;
- if (cuda_source_.length() != 0) {
- return cuda_source_;
- } else {
- if (fmt_ == "ptx") return data_;
- return "";
+ // For known compiled formats, return code as string when format matches.
+ if (format == fmt_) {
+ return ffi::String(code_.data(), code_.size());
+ }
+ // Look up the source map for an exact match (e.g. "cuda" returns the
+ // original C++ source, populated by codegen at construction time).
+ if (auto it = source_.find(format); it != source_.end()) {
+ return (*it).second;
+ }
+ // Empty-format (`mod.get_source()`) — prefer the `cuda` source if present,
+ // else fall back to code-as-string when fmt_ is textual.
+ if (format.empty()) {
+ if (auto it = source_.find("cuda"); it != source_.end()) {
+ return (*it).second;
+ }
+ if (fmt_ == "ptx" || fmt_ == "cuda") {
+ return ffi::String(code_.data(), code_.size());
+ }
}
+ return ffi::String();
}
// get a CUfunction from primary context in device_id
@@ -117,7 +126,7 @@ class CUDAModuleNode : public ffi::ModuleObj {
std::lock_guard<std::mutex> lock(mutex_);
// must recheck under the lock scope
if (module_[device_id] == nullptr) {
- CUDA_DRIVER_CALL(cuModuleLoadData(&(module_[device_id]), data_.c_str()));
+ CUDA_DRIVER_CALL(cuModuleLoadData(&(module_[device_id]), code_.data()));
static auto nvshmem_init_hook = ffi::Function::GetGlobal("runtime.nvshmem.cumodule_init");
if (nvshmem_init_hook.has_value()) {
(*nvshmem_init_hook)(static_cast<void*>(module_[device_id]));
@@ -134,15 +143,45 @@ class CUDAModuleNode : public ffi::ModuleObj {
return func;
}
+ /*!
+ * \brief JIT-compile raw CUDA C++ source to PTX/cubin/fatbin via the Python
+ * compile callback. Called from BOTH the
+ * "ffi.Module.create.cuda" lambda (when the codegen hands us
+ * fmt=="cuda") AND from LoadFromBytes (when the saved-on-disk fmt is
+ * "cuda" — the cross-compile receiver path).
+ *
+ * \param source Raw CUDA C++ source (text).
+ * \return Compiled binary bytes. Determination of the compiled format
+ * (ptx vs cubin vs fatbin) is left to the caller (heuristic on
+ * first byte: '/' → ptx-text, otherwise binary).
+ */
+ static ffi::Bytes JitCompileFromSource(const ffi::String& source) {
+ // Registry: "tvm_callback_cuda_compile" — Python-side nvcc/nvrtc wrapper.
+ // Grep hint: grep -rn 'tvm_callback_cuda_compile' src/ python/
+ auto fcompile = ffi::Function::GetGlobal("tvm_callback_cuda_compile");
+ TVM_FFI_CHECK(fcompile.has_value(), RuntimeError)
+ << "fmt=='cuda' requires tvm_callback_cuda_compile to be registered. "
+ << "Import tvm.contrib.nvcc.";
+ return (*fcompile)(source).cast<ffi::Bytes>();
+ }
+
+ /*! \brief Pick the compiled format from the JIT output's first byte. */
+ static ffi::String DetermineCompiledFormat(const ffi::Bytes& compiled) {
+ if (compiled.size() > 0 && compiled.data()[0] == '/') {
+ return ffi::String("ptx");
+ }
+ return ffi::String("cubin");
+ }
+
private:
- // the binary data
- std::string data_;
- // The format
- std::string fmt_;
+ // The binary data (compiled PTX/cubin/fatbin, or raw CUDA source if fmt == "cuda").
+ ffi::Bytes code_;
+ // The format of code_.
+ ffi::String fmt_;
// function information table.
ffi::Map<ffi::String, FunctionInfo> fmap_;
- // The cuda source.
- std::string cuda_source_;
+ // In-memory source map for InspectSource — never serialized.
+ ffi::Map<ffi::String, ffi::String> source_;
// the internal modules per GPU, to be lazily initialized.
std::array<CUmodule, kMaxNumGPUs> module_;
// internal mutex when updating the module
@@ -219,7 +258,7 @@ class CUDAWrappedFunc {
<< " grid=(" << wl.grid_dim(0) << "," << wl.grid_dim(1) << "," << wl.grid_dim(2) << "), "
<< " block=(" << wl.block_dim(0) << "," << wl.block_dim(1) << "," << wl.block_dim(2)
<< ")\n";
- std::string cuda = m_->InspectSource("");
+ ffi::String cuda = m_->InspectSource("");
if (cuda.length() != 0) {
os << "// func_name=" << func_name_ << "\n"
<< "// CUDA Source\n"
@@ -255,47 +294,56 @@ ffi::Optional<ffi::Function> CUDAModuleNode::GetFunction(const ffi::String& name
return PackFuncVoidAddr(f, info->arg_types, info->arg_extra_tags);
}
-static ffi::Module CUDAModuleCreateImpl(std::string data, std::string fmt,
+// Construct a CUDAModuleNode from in-memory payload. When fmt == "cuda" the
+// code is raw CUDA C++ source — JIT-compile via the Python callback, then
+// re-tag with the resulting compiled format ("ptx" / "cubin").
+static ffi::Module CUDAModuleCreateImpl(ffi::Bytes code, ffi::String fmt,
ffi::Map<ffi::String, FunctionInfo> fmap,
- std::string cuda_source) {
- auto n = ffi::make_object<CUDAModuleNode>(data, fmt, fmap, cuda_source);
+ ffi::Map<ffi::String, ffi::String> source) {
+ if (fmt == "cuda") {
+ // Stash the CUDA source for InspectSource before we replace `code` with
+ // the JIT output.
+ if (source.find("cuda") == source.end()) {
+ source.Set("cuda", ffi::String(code.data(), code.size()));
+ }
+ ffi::Bytes compiled =
+ CUDAModuleNode::JitCompileFromSource(ffi::String(code.data(), code.size()));
+ fmt = CUDAModuleNode::DetermineCompiledFormat(compiled);
+ code = std::move(compiled);
+ }
+ auto n = ffi::make_object<CUDAModuleNode>(code, fmt, fmap, source);
return ffi::Module(n);
}
-// Load module from module.
-ffi::Module CUDAModuleLoadFile(const std::string& file_name, const ffi::String& format) {
- std::string data;
- ffi::Map<ffi::String, FunctionInfo> fmap;
- std::string fmt = GetFileFormat(file_name, format);
- std::string meta_file = GetMetaFilePath(file_name);
- LoadBinaryFromFile(file_name, &data);
- LoadMetaDataFromFile(meta_file, &fmap);
- return CUDAModuleCreateImpl(data, fmt, fmap, std::string());
-}
-
-ffi::Module CUDAModuleLoadFromBytes(const ffi::Bytes& bytes) {
+static ffi::Module CUDAModuleLoadFromBytes(const ffi::Bytes& bytes) {
support::BytesInStream stream(bytes);
- std::string data;
+ ffi::String fmt;
ffi::Map<ffi::String, FunctionInfo> fmap;
- std::string fmt;
+ ffi::Bytes code;
stream.Read(&fmt);
TVM_FFI_ICHECK(stream.Read(&fmap));
- stream.Read(&data);
- return CUDAModuleCreateImpl(data, fmt, fmap, std::string());
+ stream.Read(&code);
+ // Source map is not serialized — it is lost on save/load round-trip.
+ // If the receiver wants InspectSource("cuda") to work, the saved bytes must
+ // have been written with fmt=="cuda" so the JIT path below re-stuffs the
+ // source map with the original C++ source.
+ return CUDAModuleCreateImpl(std::move(code), std::move(fmt), std::move(fmap),
+ ffi::Map<ffi::String, ffi::String>());
}
TVM_FFI_STATIC_INIT_BLOCK() {
namespace refl = tvm::ffi::reflection;
+ // Registry: "ffi.Module.create.cuda" — codegen-time CUDA module factory.
+ // Used by src/target/cuda/cuda_fallback_module.h:CUDAModuleCreateWithFallback.
+ // Registry: "ffi.Module.load_from_bytes.cuda" — disk loader. Only this
+ // (real) module registers a loader; the fallback module is codegen-time only.
refl::GlobalDef()
- .def("ffi.Module.load_from_file.cuda", CUDAModuleLoadFile)
- .def("ffi.Module.load_from_file.ptx", CUDAModuleLoadFile)
- .def("ffi.Module.load_from_file.cubin", CUDAModuleLoadFile)
.def("ffi.Module.load_from_bytes.cuda", CUDAModuleLoadFromBytes)
.def("ffi.Module.create.cuda",
- [](ffi::String data, ffi::String fmt, ffi::Map<ffi::String, FunctionInfo> fmap,
- ffi::String cuda_source) {
- return CUDAModuleCreateImpl(std::string(data), std::string(fmt), fmap,
- std::string(cuda_source));
+ [](ffi::Bytes code, ffi::String fmt, ffi::Map<ffi::String, FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source) {
+ return CUDAModuleCreateImpl(std::move(code), std::move(fmt), std::move(fmap),
+ std::move(source));
});
}
} // namespace runtime
diff --git a/src/runtime/cuda/cuda_module.h b/src/runtime/cuda/cuda_module.h
deleted file mode 100644
index 1bd94332ef..0000000000
--- a/src/runtime/cuda/cuda_module.h
+++ /dev/null
@@ -1,66 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-/*!
- * \file cuda_module.h
- * \brief Execution handling of CUDA kernels
- */
-#ifndef TVM_RUNTIME_CUDA_CUDA_MODULE_H_
-#define TVM_RUNTIME_CUDA_CUDA_MODULE_H_
-
-#include <tvm/ffi/extra/module.h>
-#include <tvm/ffi/function.h>
-#include <tvm/runtime/base.h>
-
-#include <memory>
-#include <string>
-#include <vector>
-
-#include "../metadata.h"
-
-namespace tvm {
-namespace runtime {
-
-/*! \brief Maximum number of GPU supported in CUDAModule */
-static constexpr const int kMaxNumGPUs = 32;
-
-/*!
- * \brief create a CUDA module from data.
- *
- * \param data The module data, can be ptx, cubin
- * \param fmt The format of the data, can be "ptx", "cubin"
- * \param fmap The map function information map of each function.
- * \param cuda_source Optional, CUDA source file
- *
- * Dispatches through the FFI registry ("ffi.Module.create.cuda").
- * Requires libtvm_runtime built with USE_CUDA=ON to have registered the creator.
- */
-inline ffi::Module CUDAModuleCreate(ffi::String data, ffi::String fmt,
- ffi::Map<ffi::String, FunctionInfo> fmap,
- ffi::String cuda_source) {
- static const auto fcreate = ffi::Function::GetGlobal("ffi.Module.create.cuda");
- TVM_FFI_CHECK(fcreate.has_value(), RuntimeError)
- << "ffi.Module.create.cuda is not registered in runtime. "
- << "Link or load libtvm_runtime built with USE_CUDA=ON.";
- return (*fcreate)(data, fmt, fmap, cuda_source).cast<ffi::Module>();
-}
-
-} // namespace runtime
-} // namespace tvm
-#endif // TVM_RUNTIME_CUDA_CUDA_MODULE_H_
diff --git a/src/runtime/file_utils.cc b/src/runtime/file_utils.cc
index dbefd9a752..095b0288ba 100644
--- a/src/runtime/file_utils.cc
+++ b/src/runtime/file_utils.cc
@@ -222,9 +222,26 @@ TVM_FFI_STATIC_INIT_BLOCK() {
SaveParams(&strm, params);
})
.def("runtime.LoadParams", [](const ffi::Bytes& s) { return ::tvm::runtime::LoadParams(s); })
- .def("runtime.LoadParamsFromFile", [](const ffi::String& path) {
- tvm::runtime::SimpleBinaryFileStream strm(path, "rb");
- return LoadParams(&strm);
+ .def("runtime.LoadParamsFromFile",
+ [](const ffi::String& path) {
+ tvm::runtime::SimpleBinaryFileStream strm(path, "rb");
+ return LoadParams(&strm);
+ })
+ // Registry: "runtime.LoadMetaDataFromJSON" — parse a tvm_meta.json
+ // string into Map<String, FunctionInfo>. Used by Python callers that
+ // build FunctionInfo maps in-memory (e.g. tirx external_kernel) without
+ // a disk round-trip.
+ .def("runtime.LoadMetaDataFromJSON", [](const ffi::String& json_str) {
+ namespace json = ::tvm::ffi::json;
+ ffi::Map<ffi::String, FunctionInfo> fmap;
+ auto root = json::Parse(std::string(json_str)).cast<json::Object>();
+ auto func_info_obj = root.at("func_info").cast<json::Object>();
+ for (const auto& kv : func_info_obj) {
+ auto info_node = ffi::make_object<FunctionInfoObj>();
+ info_node->LoadFromJSON(kv.second.cast<json::Object>());
+ fmap.Set(kv.first.cast<ffi::String>(), FunctionInfo(std::move(info_node)));
+ }
+ return fmap;
});
}
diff --git a/src/runtime/hexagon/hexagon_module.cc b/src/runtime/hexagon/hexagon_module.cc
index 5546e31b8a..f0f801ef45 100644
--- a/src/runtime/hexagon/hexagon_module.cc
+++ b/src/runtime/hexagon/hexagon_module.cc
@@ -19,94 +19,137 @@
/*!
* \file hexagon_module.cc
- * \brief The HexagonModuleNode
+ * \brief HexagonModuleNode — runtime-side, plugin-only. Reachable from C++
+ * only through the FFI registry keys "ffi.Module.create.hexagon" and
+ * "ffi.Module.load_from_bytes.hexagon". No exported header —
+ * codegen-side construction goes through
+ * src/target/hexagon/hexagon_fallback_module.h.
+ *
+ * This carrier holds the linked Hexagon `.so` blob in memory. The
+ * existing HexagonModuleNode does not perform `dlopen` (Hexagon
+ * execution happens via a separate RPC-based runtime path); kernel
+ * launches via `GetFunction` are unimplemented. The module exists
+ * primarily for cross-compile shipment (SaveToBytes) and source
+ * inspection.
*/
-#include "hexagon_module.h"
-
+#include <tvm/ffi/container/variant.h>
#include <tvm/ffi/extra/module.h>
#include <tvm/ffi/function.h>
#include <tvm/ffi/reflection/registry.h>
-#include <tvm/support/io.h>
#include <string>
#include <utility>
#include <vector>
#include "../../support/bytes_io.h"
-#include "../file_utils.h"
+#include "../metadata.h"
namespace tvm {
namespace runtime {
-HexagonModuleNode::HexagonModuleNode(std::string data, std::string fmt,
- ffi::Map<ffi::String, FunctionInfo> fmap, std::string asm_str,
- std::string obj_str, std::string ir_str, std::string bc_str)
- : data_(data), fmt_(fmt), fmap_(fmap), asm_(asm_str), obj_(obj_str), ir_(ir_str), bc_(bc_str) {}
+using HexagonSourceMap = ffi::Map<ffi::String, ffi::Variant<ffi::String, ffi::Bytes>>;
-ffi::Optional<ffi::Function> HexagonModuleNode::GetFunction(const ffi::String& name) {
- TVM_FFI_THROW(InternalError) << "HexagonModuleNode::GetFunction is not implemented.";
-}
+class HexagonModuleNode : public ffi::ModuleObj {
+ public:
+ HexagonModuleNode(ffi::Bytes code, ffi::String fmt, ffi::Map<ffi::String, FunctionInfo> fmap,
+ HexagonSourceMap source)
+ : code_(std::move(code)),
+ fmt_(std::move(fmt)),
+ fmap_(std::move(fmap)),
+ source_(std::move(source)) {}
-ffi::String HexagonModuleNode::InspectSource(const ffi::String& format) const {
- if (format == "s" || format == "asm") {
- return asm_;
+ const char* kind() const final { return "hexagon"; }
+
+ int GetPropertyMask() const final {
+ return ffi::Module::kBinarySerializable | ffi::Module::kRunnable;
}
- if (format == "ll") {
- return ir_;
+
+ ffi::Optional<ffi::Function> GetFunction(const ffi::String& name) final {
+ TVM_FFI_THROW(InternalError) << "HexagonModuleNode::GetFunction is not implemented.";
+ TVM_FFI_UNREACHABLE();
}
- return "";
-}
-void HexagonModuleNode::WriteToFile(const ffi::String& file_name, const ffi::String& format) const {
- std::string fmt = runtime::GetFileFormat(file_name, format);
- if (fmt == "so" || fmt == "dll" || fmt == "hexagon") {
- std::string meta_file = GetMetaFilePath(file_name);
- SaveMetaDataToFile(meta_file, fmap_);
- CopyFile(data_, file_name);
- } else if (fmt == "s" || fmt == "asm") {
- TVM_FFI_ICHECK(!asm_.empty()) << "Assembler source not available";
- SaveBinaryToFile(file_name, asm_);
- } else if (fmt == "o" || fmt == "obj") {
- TVM_FFI_ICHECK(!obj_.empty()) << "Object data not available";
- SaveBinaryToFile(file_name, obj_);
- } else if (fmt == "ll") {
- TVM_FFI_ICHECK(!ir_.empty()) << "LLVM IR source not available";
- SaveBinaryToFile(file_name, ir_);
- } else if (fmt == "bc") {
- TVM_FFI_ICHECK(!bc_.empty()) << "LLVM IR bitcode not available";
- SaveBinaryToFile(file_name, bc_);
- } else {
- TVM_FFI_THROW(InternalError) << "HexagonModuleNode::SaveToFile: unhandled format `" << fmt
- << "'";
+ ffi::String InspectSource(const ffi::String& format) const final {
+ if (format == fmt_) {
+ return ffi::String(code_.data(), code_.size());
+ }
+ if (auto it = source_.find(format); it != source_.end()) {
+ const auto& v = (*it).second;
+ if (auto s = v.template as<ffi::String>()) return *s;
+ if (auto b = v.template as<ffi::Bytes>()) return ffi::String(b->data(), b->size());
+ }
+ if (format == "asm") {
+ if (auto it = source_.find("s"); it != source_.end()) {
+ const auto& v = (*it).second;
+ if (auto s = v.template as<ffi::String>()) return *s;
+ }
+ }
+ return ffi::String();
}
-}
-ffi::Bytes HexagonModuleNode::SaveToBytes() const {
- std::string result;
- support::BytesOutStream stream(&result);
- stream.Write(fmt_);
- stream.Write(fmap_);
- stream.Write(data_);
- return ffi::Bytes(std::move(result));
-}
+ ffi::Bytes SaveToBytes() const final {
+ // Format: [fmt][fmap][code]. Source map is in-memory inspection only
+ // and is NEVER serialized — it is lost on save/load round-trip
+ // (matches upstream behavior; the receiver rebuilds source from code
+ // bytes if possible). HexagonFallbackModuleNode::SaveToBytes (in
+ // src/target/hexagon/hexagon_fallback_module.cc) MUST mirror this
+ // format byte-for-byte; see one-way comment there.
+ std::string buffer;
+ support::BytesOutStream stream(&buffer);
+ stream.Write(fmt_);
+ stream.Write(fmap_);
+ stream.Write(code_);
+ return ffi::Bytes(std::move(buffer));
+ }
-static ffi::Module HexagonModuleCreateImpl(std::string data, std::string fmt,
+ private:
+ // Linked Hexagon shared-library bytes (fmt="so").
+ ffi::Bytes code_;
+ // Format identifier ("so").
+ ffi::String fmt_;
+ // function information table.
+ ffi::Map<ffi::String, FunctionInfo> fmap_;
+ // In-memory source map for InspectSource — never serialized.
+ // Variant lets each blob (text "asm"/"s"/"ll" or binary "obj"/"bc") be
+ // passed in its natural form.
+ HexagonSourceMap source_;
+};
+
+static ffi::Module HexagonModuleCreateImpl(ffi::Bytes code, ffi::String fmt,
ffi::Map<ffi::String, FunctionInfo> fmap,
- std::string asm_str, std::string obj_str,
- std::string ir_str, std::string bc_str) {
- auto n = ffi::make_object<HexagonModuleNode>(data, fmt, fmap, asm_str, obj_str, ir_str, bc_str);
+ HexagonSourceMap source) {
+ auto n = ffi::make_object<HexagonModuleNode>(std::move(code), std::move(fmt), std::move(fmap),
+ std::move(source));
return ffi::Module(n);
}
+static ffi::Module HexagonModuleLoadFromBytes(const ffi::Bytes& bytes) {
+ support::BytesInStream stream(bytes);
+ ffi::String fmt;
+ ffi::Map<ffi::String, FunctionInfo> fmap;
+ ffi::Bytes code;
+ stream.Read(&fmt);
+ TVM_FFI_ICHECK(stream.Read(&fmap));
+ stream.Read(&code);
+ // Source map is not serialized — it is lost on save/load round-trip.
+ return HexagonModuleCreateImpl(std::move(code), std::move(fmt), std::move(fmap),
+ HexagonSourceMap());
+}
+
TVM_FFI_STATIC_INIT_BLOCK() {
namespace refl = tvm::ffi::reflection;
- refl::GlobalDef().def("ffi.Module.create.hexagon", [](ffi::String data, ffi::String fmt,
- ffi::Map<ffi::String, FunctionInfo> fmap,
- ffi::String asm_str, ffi::String obj_str,
- ffi::String ir_str, ffi::String bc_str) {
- return HexagonModuleCreateImpl(std::string(data), std::string(fmt), fmap, std::string(asm_str),
- std::string(obj_str), std::string(ir_str), std::string(bc_str));
- });
+ // Registry: "ffi.Module.create.hexagon" — codegen-time Hexagon module factory.
+ // Used by src/target/hexagon/hexagon_fallback_module.h:HexagonModuleCreateWithFallback.
+ // Registry: "ffi.Module.load_from_bytes.hexagon" — disk loader. Only
+ // this (real) module registers a loader; the fallback is codegen-only.
+ refl::GlobalDef()
+ .def("ffi.Module.load_from_bytes.hexagon", HexagonModuleLoadFromBytes)
+ .def("ffi.Module.create.hexagon",
+ [](ffi::Bytes code, ffi::String fmt, ffi::Map<ffi::String, FunctionInfo> fmap,
+ HexagonSourceMap source) {
+ return HexagonModuleCreateImpl(std::move(code), std::move(fmt), std::move(fmap),
+ std::move(source));
+ });
}
} // namespace runtime
diff --git a/src/runtime/hexagon/hexagon_module.h b/src/runtime/hexagon/hexagon_module.h
deleted file mode 100644
index eeae7b32b5..0000000000
--- a/src/runtime/hexagon/hexagon_module.h
+++ /dev/null
@@ -1,95 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-#ifndef TVM_RUNTIME_HEXAGON_HEXAGON_MODULE_H_
-#define TVM_RUNTIME_HEXAGON_HEXAGON_MODULE_H_
-
-#include <tvm/ffi/extra/module.h>
-#include <tvm/ffi/function.h>
-#include <tvm/runtime/logging.h>
-
-#include <array>
-#include <memory>
-#include <set>
-#include <string>
-
-#include "../metadata.h"
-
-namespace tvm {
-namespace runtime {
-
-/*!
- * \brief Create a Hexagon module from data.
- * \param data The module data.
- * \param fmt The format of the data, can be "obj".
- * \param fmap The function information map of each function.
- * \param asm_str String with the generated assembly source.
- * \param obj_str String with the object file data.
- * \param ir_str String with the disassembled LLVM IR source.
- * \param bc_str String with the bitcode LLVM IR.
- *
- * Dispatches through the FFI registry ("ffi.Module.create.hexagon").
- * Requires libtvm_runtime built with USE_HEXAGON=ON to have registered the creator.
- */
-inline ffi::Module HexagonModuleCreate(ffi::String data, ffi::String fmt,
- ffi::Map<ffi::String, FunctionInfo> fmap,
- ffi::String asm_str, ffi::String obj_str, ffi::String ir_str,
- ffi::String bc_str) {
- static const auto fcreate = ffi::Function::GetGlobal("ffi.Module.create.hexagon");
- TVM_FFI_CHECK(fcreate.has_value(), RuntimeError)
- << "ffi.Module.create.hexagon is not registered in runtime. "
- << "Link or load libtvm_runtime built with USE_HEXAGON=ON.";
- return (*fcreate)(data, fmt, fmap, asm_str, obj_str, ir_str, bc_str).cast<ffi::Module>();
-}
-
-/*!
- \brief Module implementation for compiled Hexagon binaries. It is suitable
- for managing cross-compiled Hexagon code on a host machine.
- See docstring for HexagonModuleCreate for
- construction parameter details.
- */
-class HexagonModuleNode : public ffi::ModuleObj {
- public:
- HexagonModuleNode(std::string data, std::string fmt, ffi::Map<ffi::String, FunctionInfo> fmap,
- std::string asm_str, std::string obj_str, std::string ir_str,
- std::string bc_str);
- ffi::Optional<ffi::Function> GetFunction(const ffi::String& name) final;
- ffi::String InspectSource(const ffi::String& format) const final;
- const char* kind() const final { return "hexagon"; }
- /*! \brief Get the property of the runtime module .*/
- int GetPropertyMask() const final {
- return ffi::Module::kBinarySerializable | ffi::Module::kCompilationExportable |
- ffi::Module::kRunnable;
- }
- void WriteToFile(const ffi::String& file_name, const ffi::String& format) const final;
- ffi::Bytes SaveToBytes() const final;
-
- protected:
- std::string data_;
- std::string fmt_;
- ffi::Map<ffi::String, FunctionInfo> fmap_;
- std::string asm_;
- std::string obj_;
- std::string ir_;
- std::string bc_;
-};
-
-} // namespace runtime
-} // namespace tvm
-#endif // TVM_RUNTIME_HEXAGON_HEXAGON_MODULE_H_
diff --git a/src/runtime/metal/metal_module.h b/src/runtime/metal/metal_module.h
deleted file mode 100644
index 3f4b3965ad..0000000000
--- a/src/runtime/metal/metal_module.h
+++ /dev/null
@@ -1,61 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-/*!
- * \file metal_module.h
- * \brief Execution handling of Metal kernels
- */
-#ifndef TVM_RUNTIME_METAL_METAL_MODULE_H_
-#define TVM_RUNTIME_METAL_METAL_MODULE_H_
-
-#include <tvm/ffi/container/map.h>
-#include <tvm/ffi/extra/module.h>
-#include <tvm/ffi/function.h>
-
-#include "../metadata.h"
-
-namespace tvm {
-namespace runtime {
-/*! \brief Maximum number of GPU supported in MetalModule. */
-static constexpr const int kMetalMaxNumDevice = 32;
-
-/*!
- * \brief create a metal module from data.
- *
- * \param smap The map from name to each shader kernel (FFI-typed).
- * \param fmap The map function information map of each function.
- * \param fmt The format of the source, can be "metal" or "metallib"
- * \param source Optional, source file, concatenated for debug dump
- *
- * Dispatches through the FFI registry ("ffi.Module.create.metal").
- * Requires libtvm_runtime built with USE_METAL=ON to have registered the creator.
- */
-inline ffi::Module MetalModuleCreate(ffi::Map<ffi::String, ffi::String> smap,
- ffi::Map<ffi::String, FunctionInfo> fmap, ffi::String fmt,
- ffi::String source) {
- static const auto fcreate = ffi::Function::GetGlobal("ffi.Module.create.metal");
- TVM_FFI_CHECK(fcreate.has_value(), RuntimeError)
- << "ffi.Module.create.metal is not registered in runtime. "
- << "Link or load libtvm_runtime built with USE_METAL=ON.";
- return (*fcreate)(smap, fmap, fmt, source).cast<ffi::Module>();
-}
-
-} // namespace runtime
-} // namespace tvm
-#endif // TVM_RUNTIME_METAL_METAL_MODULE_H_
diff --git a/src/runtime/metal/metal_module.mm b/src/runtime/metal/metal_module.mm
index 22168538e3..56680b0824 100644
--- a/src/runtime/metal/metal_module.mm
+++ b/src/runtime/metal/metal_module.mm
@@ -18,19 +18,23 @@
*/
/*!
- * \file metal_module.cc
+ * \file metal_module.mm
+ * \brief MetalModuleNode — runtime-side, plugin-only. Reachable from C++
+ * only through the FFI registry keys "ffi.Module.create.metal" and
+ * "ffi.Module.load_from_bytes.metal". No exported header — codegen-
+ * side construction goes through src/target/metal/metal_fallback_module.h.
*/
-#include "metal_module.h"
-#include <tvm/support/io.h>
-
#include <tvm/ffi/extra/json.h>
#include <tvm/ffi/extra/module.h>
#include <tvm/ffi/function.h>
#include <tvm/ffi/reflection/registry.h>
+#include <tvm/support/io.h>
#include <array>
#include <mutex>
#include <sstream>
#include <string>
+#include <unordered_map>
+#include <utility>
#include "../../support/bytes_io.h"
#include "../file_utils.h"
#include "../metadata.h"
@@ -41,20 +45,27 @@
namespace tvm {
namespace runtime {
-// The version of metal module
-// for future compatibility checking
-// bump when we change the binary format.
-static constexpr const char* kMetalModuleVersion = "0.1.0";
+/*! \brief Maximum number of GPU supported in MetalModule. */
+static constexpr const int kMetalMaxNumDevice = 32;
// Module to support thread-safe multi-GPU execution.
// The runtime will contain a per-device module table
// The modules will be lazily loaded
class MetalModuleNode final : public ffi::ModuleObj {
public:
- explicit MetalModuleNode(std::unordered_map<std::string, std::string> smap,
- ffi::Map<ffi::String, FunctionInfo> fmap, std::string fmt,
- std::string source)
- : smap_(smap), fmap_(fmap), fmt_(fmt), source_(source) {}
+ // Unified factory signature shared with the codegen-side fallback in
+ // src/target/metal/metal_fallback_module.h. The per-kernel `smap`
+ // payload is Map<String, Bytes> regardless of whether the format is
+ // text MSL ("metal") or compiled metallib ("metallib") — text vs binary
+ // distinction lives in `fmt`.
+ MetalModuleNode(ffi::Map<ffi::String, ffi::Bytes> smap, ffi::String fmt,
+ ffi::Map<ffi::String, FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source)
+ : smap_(std::move(smap)),
+ fmt_(std::move(fmt)),
+ fmap_(std::move(fmap)),
+ source_(std::move(source)) {}
+
const char* kind() const final { return "metal"; }
/*! \brief Get the property of the runtime module. */
@@ -64,23 +75,29 @@ class MetalModuleNode final : public ffi::ModuleObj {
ffi::Optional<ffi::Function> GetFunction(const ffi::String& name) final;
- void WriteToFile(const ffi::String& file_name, const ffi::String& format) const final {
- LOG(FATAL) << "Do not support save to file, use save to binary and export instead";
- }
-
ffi::Bytes SaveToBytes() const final {
+ // 3 fields [fmt][fmap][smap]. Source map is in-memory inspection only
+ // and is NEVER serialized — matches the cross-backend rule.
+ // MetalFallbackModuleNode::SaveToBytes (in
+ // src/target/metal/metal_fallback_module.cc) MUST mirror this format
+ // byte-for-byte; see one-way comment there.
std::string result;
support::BytesOutStream stream(&result);
- std::string version = kMetalModuleVersion;
- stream.Write(version);
- stream.Write(smap_);
- stream.Write(fmap_);
stream.Write(fmt_);
+ stream.Write(fmap_);
+ stream.Write(smap_);
return ffi::Bytes(std::move(result));
}
ffi::String InspectSource(const ffi::String& format) const final {
- // return text source if available.
- return source_;
+ if (auto it = source_.find(format); it != source_.end()) {
+ return (*it).second;
+ }
+ if (format.empty()) {
+ if (auto it = source_.find("metal"); it != source_.end()) {
+ return (*it).second;
+ }
+ }
+ return ffi::String();
}
// get a from primary context in device_id
@@ -101,17 +118,19 @@ class MetalModuleNode final : public ffi::ModuleObj {
auto kernel = smap_.find(func_name);
// Directly lookup kernels
TVM_FFI_ICHECK(kernel != smap_.end());
- const std::string& source = kernel->second;
+ const ffi::Bytes& source = (*kernel).second;
if (fmt_ == "metal") {
MTLCompileOptions* opts = [MTLCompileOptions alloc];
opts.languageVersion = MTLLanguageVersion2_3;
opts.fastMathEnabled = YES;
// opts = nil;
- lib =
- [w->devices[device_id] newLibraryWithSource:[NSString stringWithUTF8String:source.c_str()]
- options:opts
- error:&err_msg];
+ // Per-kernel payload is bytes; treat as UTF-8 MSL source.
+ std::string source_str(source.data(), source.size());
+ lib = [w->devices[device_id]
+ newLibraryWithSource:[NSString stringWithUTF8String:source_str.c_str()]
+ options:opts
+ error:&err_msg];
[opts dealloc];
if (lib == nil) {
LOG(FATAL) << "Fail to compile metal source:"
@@ -123,7 +142,7 @@ class MetalModuleNode final : public ffi::ModuleObj {
} else {
// Build from library.
auto q = dispatch_queue_create("q", DISPATCH_QUEUE_SERIAL);
- auto data = dispatch_data_create(source.c_str(), source.length(), q,
+ auto data = dispatch_data_create(source.data(), source.size(), q,
^{
});
lib = [w->devices[device_id] newLibraryWithData:data error:&err_msg];
@@ -161,14 +180,15 @@ class MetalModuleNode final : public ffi::ModuleObj {
}
}
};
- // the source shader data, can be mtl or binary
- std::unordered_map<std::string, std::string> smap_;
+ // Per-kernel payload: kernel-name -> bytes (MSL source for fmt="metal" /
+ // metallib blob for fmt="metallib").
+ ffi::Map<ffi::String, ffi::Bytes> smap_;
+ // The format ("metal" source / "metallib" compiled).
+ ffi::String fmt_;
// function information table.
ffi::Map<ffi::String, FunctionInfo> fmap_;
- // The format
- std::string fmt_;
- // The source
- std::string source_;
+ // In-memory source map for InspectSource — never serialized.
+ ffi::Map<ffi::String, ffi::String> source_;
// function information.
std::vector<DeviceEntry> finfo_;
// internal mutex when updating the module
@@ -272,62 +292,43 @@ ffi::Optional<ffi::Function> MetalModuleNode::GetFunction(const ffi::String& nam
return ret;
}
-static ffi::Module MetalModuleCreateImpl(std::unordered_map<std::string, std::string> smap,
- ffi::Map<ffi::String, FunctionInfo> fmap, std::string fmt,
- std::string source) {
+static ffi::Module MetalModuleCreateImpl(ffi::Map<ffi::String, ffi::Bytes> smap, ffi::String fmt,
+ ffi::Map<ffi::String, FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source) {
ObjectPtr<MetalModuleNode> n;
- AUTORELEASEPOOL { n = ffi::make_object<MetalModuleNode>(smap, fmap, fmt, source); };
+ AUTORELEASEPOOL {
+ n = ffi::make_object<MetalModuleNode>(std::move(smap), std::move(fmt), std::move(fmap),
+ std::move(source));
+ };
return ffi::Module(n);
}
-TVM_FFI_STATIC_INIT_BLOCK() {
- namespace refl = tvm::ffi::reflection;
- refl::GlobalDef().def(
- "runtime.module.create_metal_module",
- [](ffi::Map<ffi::String, ffi::String> smap, std::string fmap_json, std::string fmt,
- std::string source) {
- namespace json = ::tvm::ffi::json;
- auto parsed = json::Parse(fmap_json).cast<json::Object>();
- ffi::Map<ffi::String, FunctionInfo> fmap;
- for (const auto& kv : parsed) {
- auto info_node = ffi::make_object<FunctionInfoObj>();
- info_node->LoadFromJSON(kv.second.cast<json::Object>());
- fmap.Set(kv.first.cast<ffi::String>(), FunctionInfo(std::move(info_node)));
- }
-
- return MetalModuleCreateImpl(
- std::unordered_map<std::string, std::string>(smap.begin(), smap.end()), fmap, fmt,
- source);
- });
-}
-
-ffi::Module MetalModuleLoadFromBytes(const ffi::Bytes& bytes) {
+static ffi::Module MetalModuleLoadFromBytes(const ffi::Bytes& bytes) {
support::BytesInStream stream(bytes);
- // version is reserved for future changes and
- // is discarded for now
- std::string ver;
- std::unordered_map<std::string, std::string> smap;
+ ffi::String fmt;
ffi::Map<ffi::String, FunctionInfo> fmap;
- std::string fmt;
-
- stream.Read(&ver);
- stream.Read(&smap);
- TVM_FFI_ICHECK(stream.Read(&fmap));
+ ffi::Map<ffi::String, ffi::Bytes> smap;
stream.Read(&fmt);
-
- return MetalModuleCreateImpl(smap, fmap, fmt, "");
+ TVM_FFI_ICHECK(stream.Read(&fmap));
+ stream.Read(&smap);
+ // Source map is not serialized — reconstructed empty on load.
+ return MetalModuleCreateImpl(std::move(smap), std::move(fmt), std::move(fmap),
+ ffi::Map<ffi::String, ffi::String>());
}
TVM_FFI_STATIC_INIT_BLOCK() {
namespace refl = tvm::ffi::reflection;
+ // Registry: "ffi.Module.create.metal" — codegen-time Metal module factory.
+ // Used by src/target/metal/metal_fallback_module.h:MetalModuleCreateWithFallback.
+ // Registry: "ffi.Module.load_from_bytes.metal" — disk loader. Only this
+ // (real) module registers a loader; the fallback is codegen-only.
refl::GlobalDef()
.def("ffi.Module.load_from_bytes.metal", MetalModuleLoadFromBytes)
.def("ffi.Module.create.metal",
- [](ffi::Map<ffi::String, ffi::String> smap, ffi::Map<ffi::String, FunctionInfo> fmap,
- ffi::String fmt, ffi::String source) {
- return MetalModuleCreateImpl(
- std::unordered_map<std::string, std::string>(smap.begin(), smap.end()), fmap,
- std::string(fmt), std::string(source));
+ [](ffi::Map<ffi::String, ffi::Bytes> smap, ffi::String fmt,
+ ffi::Map<ffi::String, FunctionInfo> fmap, ffi::Map<ffi::String, ffi::String> source) {
+ return MetalModuleCreateImpl(std::move(smap), std::move(fmt), std::move(fmap),
+ std::move(source));
});
}
} // namespace runtime
diff --git a/src/runtime/opencl/opencl_common.h b/src/runtime/opencl/opencl_common.h
index 2ae389046f..a43b29d5ec 100644
--- a/src/runtime/opencl/opencl_common.h
+++ b/src/runtime/opencl/opencl_common.h
@@ -511,14 +511,17 @@ class OpenCLModuleNodeBase : public ffi::ModuleObj {
class OpenCLModuleNode : public OpenCLModuleNodeBase {
public:
- explicit OpenCLModuleNode(std::string data, std::string fmt,
- ffi::Map<ffi::String, FunctionInfo> fmap, std::string source)
- : OpenCLModuleNodeBase(fmap), data_(data), fmt_(fmt), source_(source) {}
+ explicit OpenCLModuleNode(ffi::Bytes code, ffi::String fmt,
+ ffi::Map<ffi::String, FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source)
+ : OpenCLModuleNodeBase(fmap),
+ code_(std::move(code)),
+ fmt_(std::move(fmt)),
+ source_(std::move(source)) {}
ffi::Optional<ffi::Function> GetFunction(const ffi::String& name) final;
// Return true if OpenCL program for the requested function and device was created
bool IsProgramCreated(const std::string& func_name, int device_id);
- void WriteToFile(const ffi::String& file_name, const ffi::String& format) const final;
ffi::Bytes SaveToBytes() const final;
void SetPreCompiledPrograms(const std::string& bytes);
std::string GetPreCompiledPrograms();
@@ -531,13 +534,15 @@ class OpenCLModuleNode : public OpenCLModuleNodeBase {
const std::string& func_name, const KTRefEntry& e) override;
private:
- // the binary data
- std::string data_;
- // The format
- std::string fmt_;
- // The OpenCL source.
- std::string source_;
- // parsed kernel data
+ // The single-binary code payload: for fmt=="cl" the bytes are the
+ // OpenCL C source; for fmt=="xclbin"/"awsxclbin"/"aocx" the bytes
+ // are a pre-compiled OpenCL binary.
+ ffi::Bytes code_;
+ // The format identifier ("cl" / "xclbin" / "awsxclbin" / "aocx").
+ ffi::String fmt_;
+ // In-memory source map for InspectSource — never serialized.
+ ffi::Map<ffi::String, ffi::String> source_;
+ // parsed kernel data (computed in Init from code_ when fmt_ == "cl").
std::unordered_map<std::string, std::string> parsed_kernels_;
};
diff --git a/src/runtime/opencl/opencl_module.cc b/src/runtime/opencl/opencl_module.cc
index c8ccbfa09f..3c0373fdee 100644
--- a/src/runtime/opencl/opencl_module.cc
+++ b/src/runtime/opencl/opencl_module.cc
@@ -19,14 +19,18 @@
/*!
* \file opencl_module.cc
+ * \brief Plugin-only OpenCL runtime module. Built only when
+ * USE_OPENCL=ON. No exported header — codegen-side construction
+ * goes through src/target/opencl/opencl_fallback_module.h:OpenCLModuleCreateWithFallback,
+ * which dispatches to "ffi.Module.create.opencl" registered
+ * below when this file is linked into the build.
*/
-#include "opencl_module.h"
-
#include <tvm/ffi/function.h>
#include <tvm/ffi/reflection/registry.h>
#include <tvm/support/io.h>
#include <string>
+#include <utility>
#include <vector>
#include "../../support/bytes_io.h"
@@ -160,30 +164,36 @@ ffi::Optional<ffi::Function> OpenCLModuleNodeBase::GetFunction(const ffi::String
return PackFuncVoidAddr(f, info->arg_types);
}
-void OpenCLModuleNode::WriteToFile(const ffi::String& file_name, const ffi::String& format) const {
- std::string fmt = GetFileFormat(file_name, format);
- TVM_FFI_ICHECK_EQ(fmt, fmt_) << "Can only save to format=" << fmt_;
- std::string meta_file = GetMetaFilePath(file_name);
- SaveMetaDataToFile(meta_file, fmap_);
- SaveBinaryToFile(file_name, data_);
-}
-
ffi::Bytes OpenCLModuleNode::SaveToBytes() const {
+ // NOTE: serialization format MUST remain byte-identical to
+ // target::OpenCLFallbackModuleNode::SaveToBytes in
+ // src/target/opencl/opencl_fallback_module.cc. This file is the
+ // source of truth; the fallback follows.
+ // 3 fields only — the source map is in-memory inspection material
+ // and is NEVER serialized (matches upstream behavior for all
+ // backends).
std::string result;
support::BytesOutStream stream(&result);
stream.Write(fmt_);
stream.Write(fmap_);
- stream.Write(data_);
+ stream.Write(code_);
return ffi::Bytes(std::move(result));
}
ffi::String OpenCLModuleNode::InspectSource(const ffi::String& format) const {
- if (format == fmt_) return data_;
- if (fmt_ == "cl") {
- return data_;
- } else {
- return source_;
+ if (auto it = source_.find(format); it != source_.end()) {
+ return (*it).second;
}
+ if (format.empty()) {
+ // Default: aggregated OpenCL C source dump (key "cl").
+ if (auto it = source_.find("cl"); it != source_.end()) {
+ return (*it).second;
+ }
+ if (fmt_ == "cl") {
+ return ffi::String(code_.data(), code_.size());
+ }
+ }
+ return ffi::String();
}
void OpenCLModuleNode::Init() {
@@ -203,13 +213,18 @@ void OpenCLModuleNode::Init() {
kid_map_[key] = e;
}
- // split into source artifacts for each kernel
- parsed_kernels_ = SplitKernels(InspectSource("cl"));
- TVM_FFI_ICHECK(!parsed_kernels_.empty()) << "The OpenCL module expects a kernel delimited "
- << "source from code generation, but no kernel "
- << "delimiter was found.";
- TVM_FFI_ICHECK_EQ(fmap_.size(), parsed_kernels_.size())
- << "The number of parsed kernel sources does not match the number of kernel functions";
+ // split into source artifacts for each kernel. For fmt=="cl" the
+ // code_ bytes are the OpenCL C source; for the binary formats
+ // (xclbin/awsxclbin/aocx) parsing is skipped — the binary blob is
+ // passed directly to clCreateProgramWithBinary in InstallKernel.
+ if (fmt_ == "cl") {
+ parsed_kernels_ = SplitKernels(std::string(code_.data(), code_.size()));
+ TVM_FFI_ICHECK(!parsed_kernels_.empty()) << "The OpenCL module expects a kernel delimited "
+ << "source from code generation, but no kernel "
+ << "delimiter was found.";
+ TVM_FFI_ICHECK_EQ(fmap_.size(), parsed_kernels_.size())
+ << "The number of parsed kernel sources does not match the number of kernel functions";
+ }
}
bool OpenCLModuleNode::IsProgramCreated(const std::string& func_name, int device_id) {
@@ -239,8 +254,8 @@ cl_kernel OpenCLModuleNode::InstallKernel(cl::OpenCLWorkspace* w, cl::OpenCLThre
clCreateProgramWithSource(w->contexts[platform], 1, &s, &len, &err);
OPENCL_CHECK_ERROR(err);
} else if (fmt_ == "xclbin" || fmt_ == "awsxclbin" || fmt_ == "aocx") {
- const unsigned char* s = (const unsigned char*)data_.c_str();
- size_t len = data_.length();
+ const unsigned char* s = reinterpret_cast<const unsigned char*>(code_.data());
+ size_t len = code_.size();
cl_int err;
cl_device_id dev = w->devices[device_id];
programs_[func_name][device_id] =
@@ -360,47 +375,41 @@ ffi::Optional<ffi::Function> OpenCLModuleNode::GetFunction(const ffi::String& na
return OpenCLModuleNodeBase::GetFunction(name);
}
-static ffi::Module OpenCLModuleCreateImpl(std::string data, std::string fmt,
+static ffi::Module OpenCLModuleCreateImpl(ffi::Bytes code, ffi::String fmt,
ffi::Map<ffi::String, FunctionInfo> fmap,
- std::string source) {
- auto n = ffi::make_object<OpenCLModuleNode>(data, fmt, fmap, source);
+ ffi::Map<ffi::String, ffi::String> source) {
+ auto n = ffi::make_object<OpenCLModuleNode>(std::move(code), std::move(fmt), std::move(fmap),
+ std::move(source));
n->Init();
return ffi::Module(n);
}
-// Load module from module.
-ffi::Module OpenCLModuleLoadFile(const std::string& file_name, const ffi::String& format) {
- std::string data;
- ffi::Map<ffi::String, FunctionInfo> fmap;
- std::string fmt = GetFileFormat(file_name, format);
- std::string meta_file = GetMetaFilePath(file_name);
- LoadBinaryFromFile(file_name, &data);
- LoadMetaDataFromFile(meta_file, &fmap);
- return OpenCLModuleCreateImpl(data, fmt, fmap, std::string());
-}
-
ffi::Module OpenCLModuleLoadFromBytes(const ffi::Bytes& bytes) {
support::BytesInStream stream(bytes);
- std::string data;
+ ffi::String fmt;
ffi::Map<ffi::String, FunctionInfo> fmap;
- std::string fmt;
+ ffi::Bytes code;
stream.Read(&fmt);
TVM_FFI_ICHECK(stream.Read(&fmap));
- stream.Read(&data);
- return OpenCLModuleCreateImpl(data, fmt, fmap, std::string());
+ stream.Read(&code);
+ // Source map is not serialized — reconstructed empty on load.
+ return OpenCLModuleCreateImpl(std::move(code), std::move(fmt), std::move(fmap),
+ ffi::Map<ffi::String, ffi::String>());
}
TVM_FFI_STATIC_INIT_BLOCK() {
namespace refl = tvm::ffi::reflection;
+ // Registry: "ffi.Module.create.opencl" — codegen-time OpenCL module factory.
+ // Used by src/target/opencl/opencl_fallback_module.h:OpenCLModuleCreateWithFallback.
+ // Registry: "ffi.Module.load_from_bytes.opencl" — disk loader. Only this
+ // (real) module registers a loader; the fallback is codegen-only.
refl::GlobalDef()
- .def("ffi.Module.load_from_file.cl", OpenCLModuleLoadFile)
- .def("ffi.Module.load_from_file.clbin", OpenCLModuleLoadFile)
.def("ffi.Module.load_from_bytes.opencl", OpenCLModuleLoadFromBytes)
.def("ffi.Module.create.opencl",
- [](ffi::String data, ffi::String fmt, ffi::Map<ffi::String, FunctionInfo> fmap,
- ffi::String source) {
- return OpenCLModuleCreateImpl(std::string(data), std::string(fmt), fmap,
- std::string(source));
+ [](ffi::Bytes code, ffi::String fmt, ffi::Map<ffi::String, FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source) {
+ return OpenCLModuleCreateImpl(std::move(code), std::move(fmt), std::move(fmap),
+ std::move(source));
});
}
} // namespace runtime
diff --git a/src/runtime/opencl/opencl_module.h b/src/runtime/opencl/opencl_module.h
deleted file mode 100644
index 9d16ea9231..0000000000
--- a/src/runtime/opencl/opencl_module.h
+++ /dev/null
@@ -1,92 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-/*!
- * \file opencl_module.h
- * \brief Execution handling of OPENCL kernels
- */
-#ifndef TVM_RUNTIME_OPENCL_OPENCL_MODULE_H_
-#define TVM_RUNTIME_OPENCL_OPENCL_MODULE_H_
-
-#include <tvm/ffi/container/map.h>
-#include <tvm/ffi/function.h>
-#include <tvm/runtime/base.h>
-
-#include <string>
-#include <unordered_map>
-
-#include "../../support/bytes_io.h"
-#include "../metadata.h"
-#include "../vulkan/spirv_shader.h"
-
-namespace tvm {
-namespace runtime {
-/*!
- * \brief Create a opencl module for GPU devices from data.
- *
- * \param data The module data.
- * \param fmt The format of the data, can be "clbin", "cl"
- * \param fmap The map function information map of each function.
- * \param source Generated OpenCL kernels.
- *
- * Dispatches through the FFI registry ("ffi.Module.create.opencl").
- * Requires libtvm_runtime built with USE_OPENCL=ON to have registered the creator.
- */
-inline ffi::Module OpenCLModuleCreate(ffi::String data, ffi::String fmt,
- ffi::Map<ffi::String, FunctionInfo> fmap,
- ffi::String source) {
- static const auto fcreate = ffi::Function::GetGlobal("ffi.Module.create.opencl");
- TVM_FFI_CHECK(fcreate.has_value(), RuntimeError)
- << "ffi.Module.create.opencl is not registered in runtime. "
- << "Link or load libtvm_runtime built with USE_OPENCL=ON.";
- return (*fcreate)(data, fmt, fmap, source).cast<ffi::Module>();
-}
-
-/*!
- * \brief Create a opencl module from SPIRV.
- *
- * \param shaders The map from function names to SPIRV binaries.
- * \param spirv_text The concatenated text representation of SPIRV modules.
- * \param fmap The map function information map of each function.
- *
- * Dispatches through the FFI registry ("ffi.Module.create.opencl.spirv").
- * Each SPIRVShader is serialised to ffi::Bytes before crossing the FFI boundary.
- * Requires libtvm_runtime built with USE_OPENCL=ON and TVM_ENABLE_SPIRV to have
- * registered the creator.
- */
-inline ffi::Module OpenCLModuleCreate(
- const std::unordered_map<std::string, spirv::SPIRVShader>& shaders,
- const ffi::String& spirv_text, const ffi::Map<ffi::String, FunctionInfo>& fmap) {
- static const auto fcreate = ffi::Function::GetGlobal("ffi.Module.create.opencl.spirv");
- TVM_FFI_CHECK(fcreate.has_value(), RuntimeError)
- << "ffi.Module.create.opencl.spirv is not registered in runtime. "
- << "Link or load libtvm_runtime built with USE_OPENCL=ON and TVM_ENABLE_SPIRV.";
- // Serialise each SPIRVShader to ffi::Bytes for the FFI boundary.
- ffi::Map<ffi::String, ffi::Bytes> shader_bytes;
- for (const auto& kv : shaders) {
- std::string buf;
- support::BytesOutStream strm(&buf);
- strm.Write(kv.second);
- shader_bytes.Set(kv.first, ffi::Bytes(std::move(buf)));
- }
- return (*fcreate)(shader_bytes, spirv_text, fmap).cast<ffi::Module>();
-}
-} // namespace runtime
-} // namespace tvm
-#endif // TVM_RUNTIME_OPENCL_OPENCL_MODULE_H_
diff --git a/src/runtime/opencl/opencl_module_spirv.cc b/src/runtime/opencl/opencl_module_spirv.cc
deleted file mode 100644
index 67e7b1c1d5..0000000000
--- a/src/runtime/opencl/opencl_module_spirv.cc
+++ /dev/null
@@ -1,159 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-#include <tvm/ffi/function.h>
-#include <tvm/ffi/reflection/registry.h>
-#include <tvm/support/io.h>
-
-#include <string>
-#include <vector>
-
-#include "../../support/bytes_io.h"
-#include "../spirv/spirv_shader.h"
-#include "opencl_common.h"
-#include "opencl_module.h"
-#include "source_utils.h"
-
-namespace tvm {
-namespace runtime {
-
-class OpenCLSPIRVModuleNode : public OpenCLModuleNodeBase {
- public:
- explicit OpenCLSPIRVModuleNode(const std::unordered_map<std::string, SPIRVShader>& shaders,
- const std::string& spirv_text,
- ffi::Map<ffi::String, FunctionInfo> fmap)
- : OpenCLModuleNodeBase(fmap), shaders_(shaders), spirv_text_(spirv_text) {}
-
- void WriteToFile(const ffi::String& file_name, const ffi::String& format) const final;
- ffi::Bytes SaveToBytes() const final;
- ffi::String InspectSource(const ffi::String& format) const final { return spirv_text_; }
-
- void Init() override;
- cl_kernel InstallKernel(cl::OpenCLWorkspace* w, cl::OpenCLThreadEntry* t,
- const std::string& func_name, const KTRefEntry& e) override;
-
- private:
- std::unordered_map<std::string, SPIRVShader> shaders_;
- std::string spirv_text_;
-};
-
-void OpenCLSPIRVModuleNode::WriteToFile(const ffi::String& file_name,
- const ffi::String& format) const {
- // TODO(masahi): How SPIRV binaries should be save to a file?
- TVM_FFI_THROW(InternalError) << "Not implemented.";
-}
-
-ffi::Bytes OpenCLSPIRVModuleNode::SaveToBytes() const {
- std::string result;
- support::BytesOutStream stream(&result);
- stream.Write(fmap_);
- stream.Write(shaders_);
- return ffi::Bytes(std::move(result));
-}
-
-void OpenCLSPIRVModuleNode::Init() {
- workspace_ = GetGlobalWorkspace();
- workspace_->Init();
- // initialize the kernel id, need to lock global table.
- std::lock_guard<std::mutex> lock(workspace_->mu);
- for (const auto& kv : fmap_) {
- const std::string& key = kv.first;
- KTRefEntry e;
- if (workspace_->free_kernel_ids.size() != 0) {
- e.kernel_id = workspace_->free_kernel_ids.back();
- workspace_->free_kernel_ids.pop_back();
- } else {
- e.kernel_id = workspace_->num_registered_kernels++;
- }
- e.version = workspace_->timestamp++;
- kid_map_[key] = e;
- }
-
- // zero initialize cl_program pointers for each device kernel
- for (auto& kv : shaders_) {
- programs_.insert({kv.first, std::vector<cl_program>(workspace_->devices.size(), nullptr)});
- }
-}
-
-cl_kernel OpenCLSPIRVModuleNode::InstallKernel(cl::OpenCLWorkspace* w, cl::OpenCLThreadEntry* t,
- const std::string& func_name, const KTRefEntry& e) {
- std::lock_guard<std::mutex> lock(build_lock_);
- int device_id = t->device.device_id;
- if (programs_[func_name][device_id] == nullptr) {
- auto it = shaders_.find(func_name);
- const unsigned char* s = (const unsigned char*)it->second.data.data();
- size_t len = it->second.data.size() * sizeof(uint32_t);
- cl_int err;
- cl_device_id dev = w->devices[device_id];
- auto platform = w->device_info[dev].platform_id;
- programs_[func_name][device_id] =
- clCreateProgramWithBinary(w->contexts[platform], 1, &dev, &len, &s, nullptr, &err);
- OPENCL_CHECK_ERROR(err);
-
- // build program
- err = clBuildProgram(programs_[func_name][device_id], 1, &dev, nullptr, nullptr, nullptr);
-
- if (err != CL_SUCCESS) {
- size_t len;
- std::string log;
- clGetProgramBuildInfo(programs_[func_name][device_id], dev, CL_PROGRAM_BUILD_LOG, 0, nullptr,
- &len);
- log.resize(len);
- clGetProgramBuildInfo(programs_[func_name][device_id], dev, CL_PROGRAM_BUILD_LOG, len,
- &log[0], nullptr);
- TVM_FFI_THROW(InternalError) << "OpenCL build error for device=" << dev << "\n" << log;
- }
- }
- // build kernel
- cl_int err;
- cl_kernel kernel = clCreateKernel(programs_[func_name][device_id], func_name.c_str(), &err);
- OPENCL_CHECK_ERROR(err);
- t->kernel_table[e.kernel_id].kernel = kernel;
- t->kernel_table[e.kernel_id].version = e.version;
- kernels_.push_back(kernel);
- return kernel;
-}
-
-static ffi::Module OpenCLSPIRVModuleCreateImpl(
- const std::unordered_map<std::string, SPIRVShader>& shaders, const std::string& spirv_text,
- ffi::Map<ffi::String, FunctionInfo> fmap) {
- auto n = ffi::make_object<OpenCLSPIRVModuleNode>(shaders, spirv_text, fmap);
- n->Init();
- return ffi::Module(n);
-}
-
-TVM_FFI_STATIC_INIT_BLOCK() {
- namespace refl = tvm::ffi::reflection;
- refl::GlobalDef().def("ffi.Module.create.opencl.spirv",
- [](ffi::Map<ffi::String, ffi::Bytes> shader_bytes, ffi::String spirv_text,
- ffi::Map<ffi::String, FunctionInfo> fmap) {
- std::unordered_map<std::string, SPIRVShader> shaders;
- for (const auto& kv : shader_bytes) {
- support::BytesInStream stream(kv.second);
- SPIRVShader shader;
- TVM_FFI_ICHECK(stream.Read(&shader));
- shaders[std::string(kv.first)] = shader;
- }
- return OpenCLSPIRVModuleCreateImpl(shaders, std::string(spirv_text),
- fmap);
- });
-}
-
-} // namespace runtime
-} // namespace tvm
diff --git a/src/runtime/rocm/rocm_module.cc b/src/runtime/rocm/rocm_module.cc
index 4d8eabcf13..b70db64de7 100644
--- a/src/runtime/rocm/rocm_module.cc
+++ b/src/runtime/rocm/rocm_module.cc
@@ -19,14 +19,16 @@
/*!
* \file rocm_module.cc
+ * \brief ROCMModuleNode — runtime-side, plugin-only. Reachable from C++ only
+ * through the FFI registry keys "ffi.Module.create.rocm" and
+ * "ffi.Module.load_from_bytes.hsaco" / "ffi.Module.load_from_bytes.hip".
+ * No exported header — codegen-side construction goes through
+ * src/target/rocm/rocm_fallback_module.h.
*/
-#include "rocm_module.h"
-
#include <hip/hip_runtime_api.h>
#include <tvm/ffi/extra/c_env_api.h>
#include <tvm/ffi/function.h>
#include <tvm/ffi/reflection/registry.h>
-#include <tvm/support/io.h>
#include <array>
#include <mutex>
@@ -34,7 +36,6 @@
#include <vector>
#include "../../support/bytes_io.h"
-#include "../file_utils.h"
#include "../metadata.h"
#include "../pack_args.h"
#include "../thread_storage_scope.h"
@@ -43,16 +44,18 @@
namespace tvm {
namespace runtime {
+// Maximum number of GPU supported in ROCMModule (file-local).
+static constexpr const int kMaxNumGPUs = 32;
+
// Module to support thread-safe multi-GPU execution.
// hipModule_t is a per-GPU module
// The runtime will contain a per-device module table
// The modules will be lazily loaded
class ROCMModuleNode : public ffi::ModuleObj {
public:
- explicit ROCMModuleNode(std::string data, std::string fmt,
- ffi::Map<ffi::String, FunctionInfo> fmap, std::string hip_source,
- std::string assembly)
- : data_(data), fmt_(fmt), fmap_(fmap), hip_source_(hip_source), assembly_(assembly) {
+ ROCMModuleNode(ffi::Bytes code, ffi::String fmt, ffi::Map<ffi::String, FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source)
+ : code_(code), fmt_(fmt), fmap_(fmap), source_(source) {
std::fill(module_.begin(), module_.end(), nullptr);
}
// destructor
@@ -71,35 +74,36 @@ class ROCMModuleNode : public ffi::ModuleObj {
}
ffi::Optional<ffi::Function> GetFunction(const ffi::String& name) final;
- void WriteToFile(const ffi::String& file_name, const ffi::String& format) const final {
- std::string fmt = GetFileFormat(file_name, format);
- std::string meta_file = GetMetaFilePath(file_name);
- // note: llvm and asm formats are not laodable, so we don't save them
- TVM_FFI_ICHECK_EQ(fmt, fmt_) << "Can only save to format=" << fmt_;
- SaveMetaDataToFile(meta_file, fmap_);
- SaveBinaryToFile(file_name, data_);
- }
-
ffi::Bytes SaveToBytes() const final {
+ // Format: [fmt][fmap][code]. Source map is in-memory inspection only and
+ // is NEVER serialized — it is lost on save/load round-trip (matches
+ // upstream behavior; the receiver rebuilds source from code bytes if
+ // possible). ROCmFallbackModuleNode::SaveToBytes (in
+ // src/target/rocm/rocm_fallback_module.cc) MUST mirror this format
+ // byte-for-byte; see one-way comment there.
std::string result;
support::BytesOutStream stream(&result);
stream.Write(fmt_);
stream.Write(fmap_);
- stream.Write(data_);
+ stream.Write(code_);
return ffi::Bytes(std::move(result));
}
ffi::String InspectSource(const ffi::String& format) const final {
if (format == fmt_) {
- return data_;
+ return ffi::String(code_.data(), code_.size());
}
- if (format == "llvm" || format == "") {
- return hip_source_;
+ if (auto it = source_.find(format); it != source_.end()) {
+ return (*it).second;
}
- if (format == "asm") {
- return assembly_;
+ if (format.empty() || format == "llvm") {
+ // Backward-compat: legacy returned `hip_source_` (LLVM IR text from the
+ // AMDGPU backend) for both empty-format and "llvm".
+ if (auto it = source_.find("hip"); it != source_.end()) {
+ return (*it).second;
+ }
}
- return "";
+ return ffi::String();
}
// get a CUfunction from primary context in device_id
@@ -108,7 +112,7 @@ class ROCMModuleNode : public ffi::ModuleObj {
// must recheck under the lock scope
if (module_[device_id] == nullptr) {
- ROCM_DRIVER_CALL(hipModuleLoadData(&(module_[device_id]), data_.c_str()));
+ ROCM_DRIVER_CALL(hipModuleLoadData(&(module_[device_id]), code_.data()));
}
hipFunction_t func;
hipError_t result = hipModuleGetFunction(&func, module_[device_id], func_name.c_str());
@@ -123,7 +127,7 @@ class ROCMModuleNode : public ffi::ModuleObj {
std::lock_guard<std::mutex> lock(mutex_);
// must recheck under the lock scope
if (module_[device_id] == nullptr) {
- ROCM_DRIVER_CALL(hipModuleLoadData(&(module_[device_id]), data_.c_str()));
+ ROCM_DRIVER_CALL(hipModuleLoadData(&(module_[device_id]), code_.data()));
}
hipDeviceptr_t global = nullptr;
size_t nbytes = 0;
@@ -134,16 +138,14 @@ class ROCMModuleNode : public ffi::ModuleObj {
}
private:
- // the binary data
- std::string data_;
- // The format
- std::string fmt_;
+ // The compiled binary data (hsaco).
+ ffi::Bytes code_;
+ // The format of code_ (always "hsaco" — ROCm has no source-JIT path).
+ ffi::String fmt_;
// function information table.
ffi::Map<ffi::String, FunctionInfo> fmap_;
- // The hip source.
- std::string hip_source_;
- // The gcn asm.
- std::string assembly_;
+ // In-memory source map for InspectSource — never serialized.
+ ffi::Map<ffi::String, ffi::String> source_;
// the internal modules per GPU, to be lazily initialized.
std::array<hipModule_t, kMaxNumGPUs> module_;
// internal mutex when updating the module
@@ -208,46 +210,40 @@ ffi::Optional<ffi::Function> ROCMModuleNode::GetFunction(const ffi::String& name
return PackFuncPackedArgAligned(f, info->arg_types);
}
-static ffi::Module ROCMModuleCreateImpl(std::string data, std::string fmt,
+static ffi::Module ROCMModuleCreateImpl(ffi::Bytes code, ffi::String fmt,
ffi::Map<ffi::String, FunctionInfo> fmap,
- std::string hip_source, std::string assembly) {
- auto n = ffi::make_object<ROCMModuleNode>(data, fmt, fmap, hip_source, assembly);
+ ffi::Map<ffi::String, ffi::String> source) {
+ auto n = ffi::make_object<ROCMModuleNode>(code, fmt, fmap, source);
return ffi::Module(n);
}
-ffi::Module ROCMModuleLoadFile(const std::string& file_name, const std::string& format) {
- std::string data;
- ffi::Map<ffi::String, FunctionInfo> fmap;
- std::string fmt = GetFileFormat(file_name, format);
- std::string meta_file = GetMetaFilePath(file_name);
- LoadBinaryFromFile(file_name, &data);
- LoadMetaDataFromFile(meta_file, &fmap);
- return ROCMModuleCreateImpl(data, fmt, fmap, std::string(), std::string());
-}
-
-ffi::Module ROCMModuleLoadFromBytes(const ffi::Bytes& bytes) {
+static ffi::Module ROCMModuleLoadFromBytes(const ffi::Bytes& bytes) {
support::BytesInStream stream(bytes);
- std::string data;
+ ffi::String fmt;
ffi::Map<ffi::String, FunctionInfo> fmap;
- std::string fmt;
+ ffi::Bytes code;
stream.Read(&fmt);
TVM_FFI_ICHECK(stream.Read(&fmap));
- stream.Read(&data);
- return ROCMModuleCreateImpl(data, fmt, fmap, std::string(), std::string());
+ stream.Read(&code);
+ // Source map is not serialized — it is lost on save/load round-trip.
+ return ROCMModuleCreateImpl(std::move(code), std::move(fmt), std::move(fmap),
+ ffi::Map<ffi::String, ffi::String>());
}
TVM_FFI_STATIC_INIT_BLOCK() {
namespace refl = tvm::ffi::reflection;
+ // Registry: "ffi.Module.create.rocm" — codegen-time ROCm module factory.
+ // Used by src/target/rocm/rocm_fallback_module.h:ROCmModuleCreateWithFallback.
+ // Registry: "ffi.Module.load_from_bytes.hsaco" / ".hip" — disk loaders.
+ // Only this (real) module registers a loader; the fallback is codegen-only.
refl::GlobalDef()
.def("ffi.Module.load_from_bytes.hsaco", ROCMModuleLoadFromBytes)
.def("ffi.Module.load_from_bytes.hip", ROCMModuleLoadFromBytes)
- .def("ffi.Module.load_from_file.hsaco", ROCMModuleLoadFile)
- .def("ffi.Module.load_from_file.hip", ROCMModuleLoadFile)
.def("ffi.Module.create.rocm",
- [](ffi::String data, ffi::String fmt, ffi::Map<ffi::String, FunctionInfo> fmap,
- ffi::String hip_source, ffi::String assembly) {
- return ROCMModuleCreateImpl(std::string(data), std::string(fmt), fmap,
- std::string(hip_source), std::string(assembly));
+ [](ffi::Bytes code, ffi::String fmt, ffi::Map<ffi::String, FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source) {
+ return ROCMModuleCreateImpl(std::move(code), std::move(fmt), std::move(fmap),
+ std::move(source));
});
}
} // namespace runtime
diff --git a/src/runtime/rocm/rocm_module.h b/src/runtime/rocm/rocm_module.h
deleted file mode 100644
index 666f73493f..0000000000
--- a/src/runtime/rocm/rocm_module.h
+++ /dev/null
@@ -1,66 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-/*!
- * \file rocm_module.h
- * \brief Execution handling of ROCM kernels
- */
-#ifndef TVM_RUNTIME_ROCM_ROCM_MODULE_H_
-#define TVM_RUNTIME_ROCM_ROCM_MODULE_H_
-
-#include <tvm/ffi/extra/module.h>
-#include <tvm/ffi/function.h>
-
-#include <memory>
-#include <string>
-#include <vector>
-
-#include "../metadata.h"
-
-namespace tvm {
-namespace runtime {
-
-/*! \brief Maximum number of GPU supported in ROCMModule */
-static constexpr const int kMaxNumGPUs = 32;
-
-/*!
- * \brief create a rocm module from data.
- *
- * \param data The module data, can be hsaco
- * \param fmt The format of the data, can be "hsaco"
- * \param fmap The map function information map of each function.
- * \param rocm_source Optional, rocm source file
- * \param assembly Optional, GCN assembly source
- *
- * Dispatches through the FFI registry ("ffi.Module.create.rocm").
- * Requires libtvm_runtime built with USE_ROCM=ON to have registered the creator.
- */
-inline ffi::Module ROCMModuleCreate(ffi::String data, ffi::String fmt,
- ffi::Map<ffi::String, FunctionInfo> fmap,
- ffi::String rocm_source, ffi::String assembly) {
- static const auto fcreate = ffi::Function::GetGlobal("ffi.Module.create.rocm");
- TVM_FFI_CHECK(fcreate.has_value(), RuntimeError)
- << "ffi.Module.create.rocm is not registered in runtime. "
- << "Link or load libtvm_runtime built with USE_ROCM=ON.";
- return (*fcreate)(data, fmt, fmap, rocm_source, assembly).cast<ffi::Module>();
-}
-
-} // namespace runtime
-} // namespace tvm
-#endif // TVM_RUNTIME_ROCM_ROCM_MODULE_H_
diff --git a/src/runtime/spirv/spirv_shader.h b/src/runtime/spirv/spirv_shader.h
deleted file mode 100644
index 11d578c6e2..0000000000
--- a/src/runtime/spirv/spirv_shader.h
+++ /dev/null
@@ -1,30 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-/*!
- * \file src/runtime/spirv/spirv_shader.h
- * \brief Deprecated include — SPIRVShader has moved to src/runtime/vulkan/spirv_shader.h.
- * This header is kept for backward compatibility; include the new path directly.
- */
-#ifndef TVM_RUNTIME_SPIRV_SPIRV_SHADER_H_
-#define TVM_RUNTIME_SPIRV_SPIRV_SHADER_H_
-
-#include "../vulkan/spirv_shader.h"
-
-#endif // TVM_RUNTIME_SPIRV_SPIRV_SHADER_H_
diff --git a/src/runtime/vulkan/vulkan_common.h b/src/runtime/vulkan/vulkan_common.h
index 95744707ca..d25817a2d7 100644
--- a/src/runtime/vulkan/vulkan_common.h
+++ b/src/runtime/vulkan/vulkan_common.h
@@ -38,9 +38,6 @@ namespace vulkan {
/*! \brief Maximum number of GPU supported in VulkanModule. */
static constexpr const int kVulkanMaxNumDevice = 8;
-/*! \brief TVM Vulkan binary pack magic number */
-static constexpr const int kVulkanModuleMagic = 0x02700027;
-
const int kMaxPushConstantsBytes = 128;
/*! \brief A mask used when we attach additional information to shaders */
diff --git a/src/runtime/vulkan/vulkan_module.cc b/src/runtime/vulkan/vulkan_module.cc
index bdb048adbe..78de679cb9 100644
--- a/src/runtime/vulkan/vulkan_module.cc
+++ b/src/runtime/vulkan/vulkan_module.cc
@@ -17,14 +17,22 @@
* under the License.
*/
-#include "vulkan_module.h"
-
+/*!
+ * \file vulkan_module.cc
+ * \brief Plugin-only Vulkan runtime module. Built only when USE_VULKAN=ON.
+ * No exported header — codegen-side construction goes through
+ * src/target/vulkan/vulkan_fallback_module.h:VulkanModuleCreateWithFallback,
+ * which dispatches to "ffi.Module.create.vulkan" registered below
+ * when this file is linked into the build.
+ */
#include <tvm/ffi/function.h>
#include <tvm/ffi/reflection/registry.h>
#include <tvm/support/io.h>
+#include <string>
+#include <utility>
+
#include "../../support/bytes_io.h"
-#include "../file_utils.h"
#include "spirv_shader.h"
#include "vulkan_wrapped_func.h"
@@ -34,7 +42,8 @@ namespace vulkan {
/*!
* \brief Deserialize a SPIRVShader from ffi::Bytes.
- * Format: flag (uint32_t) followed by data (vector<uint32_t>).
+ * Format: flag (uint32_t) followed by data (vector<uint32_t>) — matches
+ * the SPIRVShader::Save format in src/runtime/vulkan/spirv_shader.h.
*/
static SPIRVShader DeserializeSPIRVShader(const ffi::Bytes& bytes) {
support::BytesInStream stream(bytes);
@@ -43,54 +52,47 @@ static SPIRVShader DeserializeSPIRVShader(const ffi::Bytes& bytes) {
return shader;
}
-static ffi::Module VulkanModuleCreateImpl(std::unordered_map<std::string, SPIRVShader> smap,
+static ffi::Module VulkanModuleCreateImpl(ffi::Map<ffi::String, ffi::Bytes> smap, ffi::String fmt,
ffi::Map<ffi::String, FunctionInfo> fmap,
- std::string source) {
- auto n = ffi::make_object<VulkanModuleNode>(smap, fmap, source);
+ ffi::Map<ffi::String, ffi::String> source) {
+ // Convert Map<String, Bytes> smap → unordered_map<string, SPIRVShader>
+ // for the in-memory module. SaveToBytes will re-emit Map<String, Bytes>
+ // form so the bytes shape stays uniform across backends.
+ std::unordered_map<std::string, SPIRVShader> internal_smap;
+ for (const auto& kv : smap) {
+ internal_smap[std::string(kv.first)] = DeserializeSPIRVShader(kv.second);
+ }
+ auto n = ffi::make_object<VulkanModuleNode>(std::move(internal_smap), std::move(smap),
+ std::move(fmt), std::move(fmap), std::move(source));
return ffi::Module(n);
}
-ffi::Module VulkanModuleLoadFile(const std::string& file_name, const ffi::String& format) {
- std::string data;
- std::unordered_map<std::string, SPIRVShader> smap;
- ffi::Map<ffi::String, FunctionInfo> fmap;
- std::string fmt = GetFileFormat(file_name, format);
- std::string meta_file = GetMetaFilePath(file_name);
- LoadBinaryFromFile(file_name, &data);
- LoadMetaDataFromFile(meta_file, &fmap);
- support::BytesInStream stream(data);
- uint32_t magic;
- stream.Read(&magic);
- TVM_FFI_ICHECK_EQ(magic, kVulkanModuleMagic) << "VulkanModule Magic mismatch";
- stream.Read(&smap);
- return VulkanModuleCreateImpl(smap, fmap, "");
-}
-
-ffi::Module VulkanModuleLoadFromBytes(const ffi::Bytes& bytes) {
+static ffi::Module VulkanModuleLoadFromBytes(const ffi::Bytes& bytes) {
support::BytesInStream stream(bytes);
- std::unordered_map<std::string, SPIRVShader> smap;
-
- std::string fmt;
- stream.Read(&fmt);
+ ffi::String fmt;
ffi::Map<ffi::String, FunctionInfo> fmap;
+ ffi::Map<ffi::String, ffi::Bytes> smap;
+ stream.Read(&fmt);
TVM_FFI_ICHECK(stream.Read(&fmap));
stream.Read(&smap);
- return VulkanModuleCreateImpl(smap, fmap, "");
+ // Source map is not serialized — reconstructed empty on load.
+ return VulkanModuleCreateImpl(std::move(smap), std::move(fmt), std::move(fmap),
+ ffi::Map<ffi::String, ffi::String>());
}
TVM_FFI_STATIC_INIT_BLOCK() {
namespace refl = tvm::ffi::reflection;
+ // Registry: "ffi.Module.create.vulkan" — codegen-time Vulkan module factory.
+ // Used by src/target/vulkan/vulkan_fallback_module.h:VulkanModuleCreateWithFallback.
+ // Registry: "ffi.Module.load_from_bytes.vulkan" — disk loader. Only this
+ // (real) module registers a loader; the fallback is codegen-only.
refl::GlobalDef()
- .def("ffi.Module.load_from_file.vulkan", VulkanModuleLoadFile)
.def("ffi.Module.load_from_bytes.vulkan", VulkanModuleLoadFromBytes)
.def("ffi.Module.create.vulkan",
- [](ffi::Map<ffi::String, ffi::Bytes> shader_bytes,
- ffi::Map<ffi::String, FunctionInfo> fmap, ffi::String source) {
- std::unordered_map<std::string, SPIRVShader> smap;
- for (const auto& kv : shader_bytes) {
- smap[std::string(kv.first)] = DeserializeSPIRVShader(kv.second);
- }
- return VulkanModuleCreateImpl(smap, fmap, std::string(source));
+ [](ffi::Map<ffi::String, ffi::Bytes> smap, ffi::String fmt,
+ ffi::Map<ffi::String, FunctionInfo> fmap, ffi::Map<ffi::String, ffi::String> source) {
+ return VulkanModuleCreateImpl(std::move(smap), std::move(fmt), std::move(fmap),
+ std::move(source));
});
}
diff --git a/src/runtime/vulkan/vulkan_module.h b/src/runtime/vulkan/vulkan_module.h
deleted file mode 100644
index d8fdda4d92..0000000000
--- a/src/runtime/vulkan/vulkan_module.h
+++ /dev/null
@@ -1,74 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-#ifndef TVM_RUNTIME_VULKAN_VULKAN_MODULE_H_
-#define TVM_RUNTIME_VULKAN_VULKAN_MODULE_H_
-
-#include <tvm/ffi/container/map.h>
-#include <tvm/ffi/extra/module.h>
-#include <tvm/ffi/function.h>
-#include <tvm/runtime/base.h>
-
-#include <string>
-#include <unordered_map>
-
-#include "../../support/bytes_io.h"
-#include "../metadata.h"
-#include "spirv_shader.h"
-
-namespace tvm {
-namespace runtime {
-namespace vulkan {
-
-/*!
- * \brief Create a Vulkan module from SPIRV shaders.
- *
- * \param smap Map from function name to SPIRVShader.
- * \param fmap Map from function name to FunctionInfo.
- * \param source Optional SPIRV text (for inspection).
- *
- * Dispatches through the FFI registry ("ffi.Module.create.vulkan").
- * Each SPIRVShader is serialised to ffi::Bytes before crossing the FFI boundary
- * and rehydrated on the runtime side.
- * Requires libtvm_runtime built with USE_VULKAN=ON to have registered the creator.
- */
-inline ffi::Module VulkanModuleCreate(const std::unordered_map<std::string, SPIRVShader>& smap,
- const ffi::Map<ffi::String, FunctionInfo>& fmap,
- const ffi::String& source) {
- static const auto fcreate = ffi::Function::GetGlobal("ffi.Module.create.vulkan");
- TVM_FFI_CHECK(fcreate.has_value(), RuntimeError)
- << "ffi.Module.create.vulkan is not registered in runtime. "
- << "Link or load libtvm_runtime built with USE_VULKAN=ON.";
- // Serialise each SPIRVShader to ffi::Bytes for the FFI boundary.
- ffi::Map<ffi::String, ffi::Bytes> shader_bytes;
- for (const auto& kv : smap) {
- std::string buf;
- support::BytesOutStream strm(&buf);
- strm.Write(kv.second);
- shader_bytes.Set(kv.first, ffi::Bytes(std::move(buf)));
- }
- return (*fcreate)(shader_bytes, fmap, source).cast<ffi::Module>();
-}
-
-} // namespace vulkan
-
-using vulkan::VulkanModuleCreate;
-} // namespace runtime
-} // namespace tvm
-#endif // TVM_RUNTIME_VULKAN_VULKAN_MODULE_H_
diff --git a/src/runtime/vulkan/vulkan_wrapped_func.cc b/src/runtime/vulkan/vulkan_wrapped_func.cc
index 952e546fdd..2b802e0a85 100644
--- a/src/runtime/vulkan/vulkan_wrapped_func.cc
+++ b/src/runtime/vulkan/vulkan_wrapped_func.cc
@@ -24,7 +24,6 @@
#include <utility>
#include "../../support/bytes_io.h"
-#include "../file_utils.h"
#include "vulkan_device_api.h"
namespace tvm {
@@ -232,8 +231,8 @@ std::shared_ptr<VulkanPipeline> VulkanModuleNode::GetPipeline(size_t device_id,
auto pe = std::make_shared<VulkanPipeline>();
{
// create shader
- auto sit = smap_.find(func_name);
- TVM_FFI_ICHECK(sit != smap_.end());
+ auto sit = internal_smap_.find(func_name);
+ TVM_FFI_ICHECK(sit != internal_smap_.end());
pe->use_ubo = sit->second.flag & (1 << ShaderMetaDataFlagMask::kUseUBO);
const std::vector<uint32_t>& data = sit->second.data;
VkShaderModuleCreateInfo shader_cinfo;
@@ -405,20 +404,13 @@ std::shared_ptr<VulkanPipeline> VulkanModuleNode::GetPipeline(size_t device_id,
return pe;
}
-void VulkanModuleNode::WriteToFile(const ffi::String& file_name, const ffi::String& format) const {
- std::string fmt = GetFileFormat(file_name, format);
- TVM_FFI_ICHECK_EQ(fmt, fmt_) << "Can only save to customized format vulkan";
- std::string meta_file = GetMetaFilePath(file_name);
- SaveMetaDataToFile(meta_file, fmap_);
- std::string result;
- support::BytesOutStream stream(&result);
- uint32_t magic = kVulkanModuleMagic;
- stream.Write(magic);
- stream.Write(smap_);
- SaveBinaryToFile(file_name, result);
-}
-
ffi::Bytes VulkanModuleNode::SaveToBytes() const {
+ // NOTE: serialization format MUST remain byte-identical to
+ // target::VulkanFallbackModuleNode::SaveToBytes in
+ // src/target/vulkan/vulkan_fallback_module.cc. This file is the
+ // source of truth; the fallback follows.
+ // 3 fields only — the source map is in-memory inspection material and
+ // is NEVER serialized (matches upstream behavior for all backends).
std::string result;
support::BytesOutStream stream(&result);
stream.Write(fmt_);
@@ -428,8 +420,16 @@ ffi::Bytes VulkanModuleNode::SaveToBytes() const {
}
ffi::String VulkanModuleNode::InspectSource(const ffi::String& format) const {
- // can only return disassembly code.
- return source_;
+ if (auto it = source_.find(format); it != source_.end()) {
+ return (*it).second;
+ }
+ if (format.empty()) {
+ // Default: aggregated SPIR-V text dump (key "spv").
+ if (auto it = source_.find("spv"); it != source_.end()) {
+ return (*it).second;
+ }
+ }
+ return ffi::String();
}
} // namespace vulkan
diff --git a/src/runtime/vulkan/vulkan_wrapped_func.h b/src/runtime/vulkan/vulkan_wrapped_func.h
index 1b189304c2..33f9cc1205 100644
--- a/src/runtime/vulkan/vulkan_wrapped_func.h
+++ b/src/runtime/vulkan/vulkan_wrapped_func.h
@@ -29,8 +29,8 @@
#include "../metadata.h"
#include "../pack_args.h"
-#include "../spirv/spirv_shader.h"
#include "../thread_storage_scope.h"
+#include "spirv_shader.h"
#include "vulkan/vulkan_core.h"
#include "vulkan_common.h"
#include "vulkan_device.h"
@@ -82,9 +82,15 @@ class VulkanWrappedFunc {
class VulkanModuleNode final : public ffi::ModuleObj {
public:
- explicit VulkanModuleNode(std::unordered_map<std::string, SPIRVShader> smap,
- ffi::Map<ffi::String, FunctionInfo> fmap, std::string source)
- : smap_(smap), fmap_(fmap), source_(source) {}
+ explicit VulkanModuleNode(std::unordered_map<std::string, SPIRVShader> internal_smap,
+ ffi::Map<ffi::String, ffi::Bytes> smap, ffi::String fmt,
+ ffi::Map<ffi::String, FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source)
+ : internal_smap_(std::move(internal_smap)),
+ smap_(std::move(smap)),
+ fmt_(std::move(fmt)),
+ fmap_(std::move(fmap)),
+ source_(std::move(source)) {}
~VulkanModuleNode();
const char* kind() const final { return "vulkan"; }
@@ -99,20 +105,22 @@ class VulkanModuleNode final : public ffi::ModuleObj {
std::shared_ptr<VulkanPipeline> GetPipeline(size_t device_id, const std::string& func_name,
size_t num_pack_args);
- void WriteToFile(const ffi::String& file_name, const ffi::String& format) const final;
-
ffi::Bytes SaveToBytes() const final;
ffi::String InspectSource(const ffi::String& format) const final;
private:
- // function information table.
- std::unordered_map<std::string, SPIRVShader> smap_;
+ // Deserialized SPIRV shaders, used by GetPipeline at runtime.
+ std::unordered_map<std::string, SPIRVShader> internal_smap_;
+ // Per-kernel serialized SPIRVShader bytes, kept for byte-identical
+ // SaveToBytes vs target::VulkanFallbackModuleNode. Both forms carry
+ // the same shaders; internal_smap_ is the deserialized cache.
+ ffi::Map<ffi::String, ffi::Bytes> smap_;
+ // The format identifier — always "vulkan" today.
+ ffi::String fmt_;
// function information table.
ffi::Map<ffi::String, FunctionInfo> fmap_;
- // The format
- std::string fmt_{"vulkan"};
- // The source
- std::string source_;
+ // In-memory source map for InspectSource — never serialized.
+ ffi::Map<ffi::String, ffi::String> source_;
// Guards accesses to `ecache_`
std::mutex mutex_;
diff --git a/src/target/source/codegen_cuda.cc b/src/target/cuda/codegen_cuda.cc
similarity index 96%
rename from src/target/source/codegen_cuda.cc
rename to src/target/cuda/codegen_cuda.cc
index fec1c302d4..d76e5fbac1 100644
--- a/src/target/source/codegen_cuda.cc
+++ b/src/target/cuda/codegen_cuda.cc
@@ -25,6 +25,7 @@
#include <tvm/arith/analyzer.h>
#include <tvm/ffi/function.h>
+#include <tvm/ffi/reflection/registry.h>
#include <tvm/s_tir/stmt.h>
#include <tvm/tirx/index_map.h>
#include <tvm/tirx/stmt_functor.h>
@@ -35,6 +36,8 @@
#include <vector>
#include "../../tirx/transform/ir_utils.h"
+#include "../build_common.h"
+#include "cuda_fallback_module.h"
#include "literal/cuda_half_t.h"
#include "literal/cuda_int8_t.h"
#include "ptx.h"
@@ -1784,5 +1787,59 @@ void CodeGenCUDA::PrintVecElemLoadExpr(DataType t, int i, const std::string& val
return;
}
+// CUDA codegen entry point. Generates CUDA C++ source, optionally lets a
+// Python postproc hook rewrite it, and hands the source bytes off to the
+// fallback-aware module factory. The factory may JIT to PTX/cubin via
+// `tvm_callback_cuda_compile` (CUDAModuleNode::JitCompileFromSource) when
+// USE_CUDA=ON; on USE_CUDA=OFF builds (or when TVM_COMPILE_FORCE_FALLBACK is
+// set), it returns a `CUDAFallbackModuleNode` carrying the raw source for
+// later cross-compile.
+ffi::Module BuildCUDA(IRModule mod, Target target) {
+ bool output_ssa = false;
+ CodeGenCUDA cg;
+ cg.Init(output_ssa);
+
+ ffi::Map<GlobalVar, PrimFunc> functions;
+ for (auto [gvar, base_func] : mod->functions) {
+ TVM_FFI_ICHECK(base_func->IsInstance<PrimFuncNode>()) << "CodeGenCUDA: Can only take PrimFunc";
+ auto prim_func = Downcast<PrimFunc>(base_func);
+ auto calling_conv =
+ prim_func->GetAttr<Integer>(tvm::attr::kCallingConv, Integer(tvm::CallingConv::kDefault));
+ TVM_FFI_ICHECK(calling_conv == CallingConv::kDeviceKernelLaunch ||
+ calling_conv == CallingConv::kDefault)
+ << "CodeGenCUDA: expect calling_conv equals CallingConv::kDeviceKernelLaunch or "
+ "CallingConv::kDefault";
+ functions.Set(gvar, prim_func);
+ }
+
+ for (auto [gvar, prim_func] : functions) {
+ cg.DeclareFunction(gvar, prim_func);
+ }
+ for (auto [gvar, prim_func] : functions) {
+ cg.AddFunction(gvar, prim_func);
+ }
+
+ std::string code = cg.Finish();
+
+ if (auto f = ffi::Function::GetGlobal("tvm_callback_cuda_postproc")) {
+ code = (*f)(code, target).cast<std::string>();
+ }
+
+ // Hand off raw CUDA source to the fallback-aware factory. When the real
+ // CUDA runtime is registered (USE_CUDA=ON and not forced-fallback) the
+ // factory invokes JitCompileFromSource via tvm_callback_cuda_compile and
+ // builds a real CUDAModuleNode. Otherwise it stores the source in a
+ // CUDAFallbackModuleNode for later cross-compile.
+ ffi::Map<ffi::String, ffi::String> source_map;
+ return target::CUDAModuleCreateWithFallback(
+ ffi::Bytes(code.data(), code.size()), ffi::String("cuda"), ExtractFuncInfo(mod), source_map);
+}
+
+TVM_FFI_STATIC_INIT_BLOCK() {
+ namespace refl = tvm::ffi::reflection;
+ refl::GlobalDef().def("target.build.cuda", BuildCUDA);
+}
+TVM_REGISTER_PASS_CONFIG_OPTION("cuda.kernels_output_dir", ffi::String);
+
} // namespace codegen
} // namespace tvm
diff --git a/src/target/source/codegen_cuda.h b/src/target/cuda/codegen_cuda.h
similarity index 97%
rename from src/target/source/codegen_cuda.h
rename to src/target/cuda/codegen_cuda.h
index 39a15def56..54431df313 100644
--- a/src/target/source/codegen_cuda.h
+++ b/src/target/cuda/codegen_cuda.h
@@ -21,8 +21,8 @@
* \file codegen_cuda.h
* \brief Utility to generate CUDA code
*/
-#ifndef TVM_TARGET_SOURCE_CODEGEN_CUDA_H_
-#define TVM_TARGET_SOURCE_CODEGEN_CUDA_H_
+#ifndef TVM_TARGET_CUDA_CODEGEN_CUDA_H_
+#define TVM_TARGET_CUDA_CODEGEN_CUDA_H_
#include <tvm/target/codegen.h>
#include <tvm/tirx/expr.h>
@@ -31,7 +31,7 @@
#include <string>
#include <unordered_map>
-#include "codegen_c.h"
+#include "../source/codegen_c.h"
namespace tvm {
namespace codegen {
@@ -126,4 +126,4 @@ class CodeGenCUDA final : public CodeGenC {
} // namespace codegen
} // namespace tvm
-#endif // TVM_TARGET_SOURCE_CODEGEN_CUDA_H_
+#endif // TVM_TARGET_CUDA_CODEGEN_CUDA_H_
diff --git a/src/target/cuda/cuda_fallback_module.cc b/src/target/cuda/cuda_fallback_module.cc
new file mode 100644
index 0000000000..a0da7fea1e
--- /dev/null
+++ b/src/target/cuda/cuda_fallback_module.cc
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file cuda_fallback_module.cc
+ * \brief CUDAFallbackModuleNode — codegen-time placeholder used when the CUDA
+ * runtime is not linked. Mirrors `CUDAModuleNode`'s save/load format
+ * byte-for-byte; see one-way comment in `SaveToBytes` below.
+ * Always compiled (independent of USE_CUDA); never registered as an
+ * FFI factory or loader.
+ */
+#include "cuda_fallback_module.h"
+
+#include <tvm/ffi/extra/module.h>
+#include <tvm/ffi/function.h>
+
+#include <string>
+#include <utility>
+
+#include "../../support/bytes_io.h"
+
+namespace tvm {
+namespace target {
+
+class CUDAFallbackModuleNode : public ffi::ModuleObj {
+ public:
+ CUDAFallbackModuleNode(ffi::Bytes code, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source)
+ : code_(std::move(code)),
+ fmt_(std::move(fmt)),
+ fmap_(std::move(fmap)),
+ source_(std::move(source)) {}
+
+ // Mirror the real module's kind so consumers cannot distinguish at the
+ // kind/api layer. Saved bytes load back as a real CUDAModuleNode on a
+ // CUDA-equipped receiver.
+ const char* kind() const final { return "cuda"; }
+
+ int GetPropertyMask() const final { return ffi::Module::kBinarySerializable; }
+
+ ffi::Optional<ffi::Function> GetFunction(const ffi::String& name) final {
+ TVM_FFI_THROW(RuntimeError)
+ << "CUDA runtime is not linked into this build; cannot launch kernels. "
+ << "Re-link with USE_CUDA=ON or load this module in a CUDA-equipped "
+ << "environment via tvm.runtime.load_module.";
+ TVM_FFI_UNREACHABLE();
+ }
+
+ ffi::Bytes SaveToBytes() const final {
+ // NOTE: serialization format MUST remain byte-identical to
+ // CUDAModuleNode::SaveToBytes in src/runtime/cuda/cuda_module.cc (the
+ // source of truth). Both produce a kind="cuda" artifact that the loader
+ // (ffi.Module.load_from_bytes.cuda, registered only when USE_CUDA=ON)
+ // deserializes. If the real impl's format changes, mirror the change
+ // here. The dependency is one-way: this file follows; cuda_module.cc
+ // does not reference this file.
+ std::string buffer;
+ support::BytesOutStream stream(&buffer);
+ stream.Write(fmt_);
+ stream.Write(fmap_);
+ stream.Write(code_);
+ return ffi::Bytes(std::move(buffer));
+ }
+
+ ffi::String InspectSource(const ffi::String& format) const final {
+ if (format == fmt_) {
+ return ffi::String(code_.data(), code_.size());
+ }
+ if (auto it = source_.find(format); it != source_.end()) {
+ return (*it).second;
+ }
+ if (format.empty()) {
+ if (auto it = source_.find("cuda"); it != source_.end()) {
+ return (*it).second;
+ }
+ if (fmt_ == "ptx" || fmt_ == "cuda") {
+ return ffi::String(code_.data(), code_.size());
+ }
+ }
+ return ffi::String();
+ }
+
+ private:
+ // Whatever the codegen produced (raw CUDA source, PTX, cubin, fatbin).
+ ffi::Bytes code_;
+ // Format identifier ("cuda", "ptx", "cubin", "fatbin").
+ ffi::String fmt_;
+ // function information table.
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap_;
+ // In-memory source map for InspectSource — never serialized.
+ ffi::Map<ffi::String, ffi::String> source_;
+};
+
+ffi::Module CUDAFallbackModuleCreate(ffi::Bytes code, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source) {
+ auto n = ffi::make_object<CUDAFallbackModuleNode>(std::move(code), std::move(fmt),
+ std::move(fmap), std::move(source));
+ return ffi::Module(n);
+}
+
+} // namespace target
+} // namespace tvm
diff --git a/src/target/cuda/cuda_fallback_module.h b/src/target/cuda/cuda_fallback_module.h
new file mode 100644
index 0000000000..f328e09c8b
--- /dev/null
+++ b/src/target/cuda/cuda_fallback_module.h
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file cuda_fallback_module.h
+ * \brief Codegen-facing CUDA module factory.
+ *
+ * `CUDAModuleCreateWithFallback` is the ONLY entry point codegen uses to
+ * construct a CUDA `ffi::Module`. It tries the runtime-registered factory
+ * "ffi.Module.create.cuda" via the FFI registry; on miss it constructs a
+ * `CUDAFallbackModuleNode` directly. The fallback exists so that codegen
+ * can succeed on a build where the CUDA runtime is not linked. This
+ * setup is helpful for cross compilation where we compile on one env
+ * and run on another.
+ */
+#ifndef TVM_TARGET_CUDA_CUDA_FALLBACK_MODULE_H_
+#define TVM_TARGET_CUDA_CUDA_FALLBACK_MODULE_H_
+
+#include <tvm/ffi/container/map.h>
+#include <tvm/ffi/extra/module.h>
+#include <tvm/ffi/function.h>
+
+#include <utility>
+
+#include "../../runtime/metadata.h"
+#include "../../support/env.h"
+
+namespace tvm {
+namespace target {
+
+/*!
+ * \brief Construct a `CUDAFallbackModuleNode` directly (no FFI registry
+ * round-trip). Used by `CUDAModuleCreateWithFallback` and by tests
+ * that explicitly want a fallback instance.
+ */
+ffi::Module CUDAFallbackModuleCreate(ffi::Bytes code, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source);
+
+/*!
+ * \brief Codegen-time CUDA module factory. Tries the FFI-registered
+ * "ffi.Module.create.cuda"; on miss or when forced via env var,
+ * falls back to `CUDAFallbackModuleCreate`.
+ *
+ * - Registry hit (USE_CUDA=ON build) → real `CUDAModuleNode` returned.
+ * - Registry miss (USE_CUDA=OFF build) → `CUDAFallbackModuleNode` returned.
+ * - `TVM_COMPILE_FORCE_FALLBACK` env var truthy → fallback regardless of
+ * registry state (used by per-backend fallback tests on a USE_X=ON CI
+ * box).
+ */
+inline ffi::Module CUDAModuleCreateWithFallback(ffi::Bytes code, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source) {
+ if (tvm::support::GetEnv<bool>("TVM_COMPILE_FORCE_FALLBACK", false)) {
+ return CUDAFallbackModuleCreate(std::move(code), std::move(fmt), std::move(fmap),
+ std::move(source));
+ }
+ // Registry: "ffi.Module.create.cuda" — real CUDA runtime factory.
+ // Grep hint: grep -rn 'ffi.Module.create.cuda' src/
+ auto fcreate = ffi::Function::GetGlobal("ffi.Module.create.cuda");
+ if (fcreate.has_value()) {
+ return (*fcreate)(code, fmt, fmap, source).cast<ffi::Module>();
+ }
+ return CUDAFallbackModuleCreate(std::move(code), std::move(fmt), std::move(fmap),
+ std::move(source));
+}
+
+} // namespace target
+} // namespace tvm
+#endif // TVM_TARGET_CUDA_CUDA_FALLBACK_MODULE_H_
diff --git a/src/target/source/intrin_rule_cuda.cc b/src/target/cuda/intrin_rule_cuda.cc
similarity index 100%
rename from src/target/source/intrin_rule_cuda.cc
rename to src/target/cuda/intrin_rule_cuda.cc
diff --git a/src/target/source/literal/cuda_half_t.h b/src/target/cuda/literal/cuda_half_t.h
similarity index 100%
rename from src/target/source/literal/cuda_half_t.h
rename to src/target/cuda/literal/cuda_half_t.h
diff --git a/src/target/source/literal/cuda_int8_t.h b/src/target/cuda/literal/cuda_int8_t.h
similarity index 100%
rename from src/target/source/literal/cuda_int8_t.h
rename to src/target/cuda/literal/cuda_int8_t.h
diff --git a/src/target/llvm/codegen_nvptx.cc b/src/target/cuda/llvm/codegen_nvptx.cc
similarity index 95%
rename from src/target/llvm/codegen_nvptx.cc
rename to src/target/cuda/llvm/codegen_nvptx.cc
index 191874bd57..64dea29915 100644
--- a/src/target/llvm/codegen_nvptx.cc
+++ b/src/target/cuda/llvm/codegen_nvptx.cc
@@ -52,10 +52,10 @@
#include <utility>
#include <vector>
-#include "../../runtime/cuda/cuda_module.h"
-#include "../build_common.h"
-#include "codegen_llvm.h"
-#include "llvm_instance.h"
+#include "../../build_common.h"
+#include "../../llvm/codegen_llvm.h"
+#include "../../llvm/llvm_instance.h"
+#include "../cuda_fallback_module.h"
namespace tvm {
namespace codegen {
@@ -345,7 +345,13 @@ ffi::Module BuildNVPTX(IRModule mod, Target target) {
#endif
pass.run(*module);
std::string ptx(data_ptx.begin(), data_ptx.end());
- return CUDAModuleCreate(ptx, "ptx", ExtractFuncInfo(mod), ll);
+ // BuildNVPTX produces PTX directly via the LLVM AMDGPU backend; hand it to
+ // the fallback-aware factory. Source map is `{"ll": ll}` so InspectSource
+ // can recover the LLVM IR even when the receiver only has a fallback module.
+ ffi::Map<ffi::String, ffi::String> source_map;
+ source_map.Set("ll", ll);
+ return target::CUDAModuleCreateWithFallback(ffi::Bytes(ptx.data(), ptx.size()),
+ ffi::String("ptx"), ExtractFuncInfo(mod), source_map);
}
TVM_FFI_STATIC_INIT_BLOCK() {
diff --git a/src/target/source/ptx.cc b/src/target/cuda/ptx.cc
similarity index 100%
rename from src/target/source/ptx.cc
rename to src/target/cuda/ptx.cc
diff --git a/src/target/source/ptx.h b/src/target/cuda/ptx.h
similarity index 100%
rename from src/target/source/ptx.h
rename to src/target/cuda/ptx.h
diff --git a/src/target/hexagon/hexagon_fallback_module.cc b/src/target/hexagon/hexagon_fallback_module.cc
new file mode 100644
index 0000000000..3b82291fdd
--- /dev/null
+++ b/src/target/hexagon/hexagon_fallback_module.cc
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file hexagon_fallback_module.cc
+ * \brief HexagonFallbackModuleNode — codegen-time placeholder used when the
+ * Hexagon runtime is not linked. Mirrors `HexagonModuleNode`'s
+ * save/load format byte-for-byte; see one-way comment in
+ * `SaveToBytes` below. Always compiled (independent of USE_HEXAGON);
+ * never registered as an FFI factory or loader.
+ */
+#include "hexagon_fallback_module.h"
+
+#include <tvm/ffi/extra/module.h>
+#include <tvm/ffi/function.h>
+
+#include <string>
+#include <utility>
+
+#include "../../support/bytes_io.h"
+
+namespace tvm {
+namespace target {
+
+class HexagonFallbackModuleNode : public ffi::ModuleObj {
+ public:
+ HexagonFallbackModuleNode(ffi::Bytes code, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ HexagonSourceMap source)
+ : code_(std::move(code)),
+ fmt_(std::move(fmt)),
+ fmap_(std::move(fmap)),
+ source_(std::move(source)) {}
+
+ // Mirror the real module's kind so consumers cannot distinguish at the
+ // kind/api layer. Saved bytes load back as a real HexagonModuleNode on
+ // a Hexagon-equipped receiver.
+ const char* kind() const final { return "hexagon"; }
+
+ int GetPropertyMask() const final { return ffi::Module::kBinarySerializable; }
+
+ ffi::Optional<ffi::Function> GetFunction(const ffi::String& name) final {
+ TVM_FFI_THROW(RuntimeError) << "Hexagon runtime is not linked into this build; cannot launch "
+ << "kernels. Re-link with USE_HEXAGON=ON or load this module in a "
+ << "Hexagon-equipped environment via tvm.runtime.load_module.";
+ TVM_FFI_UNREACHABLE();
+ }
+
+ ffi::Bytes SaveToBytes() const final {
+ // NOTE: serialization format MUST remain byte-identical to
+ // HexagonModuleNode::SaveToBytes in src/runtime/hexagon/hexagon_module.cc
+ // (the source of truth). Both produce a kind="hexagon" artifact that
+ // the loader (ffi.Module.load_from_bytes.hexagon, registered only when
+ // USE_HEXAGON=ON) deserializes. If the real impl's format changes,
+ // mirror the change here. The dependency is one-way: this file
+ // follows; hexagon_module.cc does not reference this file.
+ //
+ // 3 fields only — the source map is in-memory inspection material and
+ // is NEVER serialized (matches upstream behavior for all backends).
+ std::string buffer;
+ support::BytesOutStream stream(&buffer);
+ stream.Write(fmt_);
+ stream.Write(fmap_);
+ stream.Write(code_);
+ return ffi::Bytes(std::move(buffer));
+ }
+
+ ffi::String InspectSource(const ffi::String& format) const final {
+ if (format == fmt_) {
+ return ffi::String(code_.data(), code_.size());
+ }
+ if (auto it = source_.find(format); it != source_.end()) {
+ const auto& v = (*it).second;
+ if (auto s = v.template as<ffi::String>()) {
+ return *s;
+ }
+ if (auto b = v.template as<ffi::Bytes>()) {
+ return ffi::String(b->data(), b->size());
+ }
+ }
+ // Backward compat: legacy HexagonModuleNode treated "asm" as alias of
+ // "s". Keep both keys discoverable.
+ if (format == "asm") {
+ if (auto it = source_.find("s"); it != source_.end()) {
+ const auto& v = (*it).second;
+ if (auto s = v.template as<ffi::String>()) return *s;
+ }
+ }
+ return ffi::String();
+ }
+
+ private:
+ // Linked Hexagon shared-library bytes (fmt="so").
+ ffi::Bytes code_;
+ // Format identifier ("so").
+ ffi::String fmt_;
+ // function information table.
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap_;
+ // In-memory source map for InspectSource — never serialized.
+ // Variant lets each upstream call site pass its blob in its natural
+ // form: text ("asm"/"s"/"ll") or binary ("obj"/"bc").
+ HexagonSourceMap source_;
+};
+
+ffi::Module HexagonFallbackModuleCreate(ffi::Bytes code, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ HexagonSourceMap source) {
+ auto n = ffi::make_object<HexagonFallbackModuleNode>(std::move(code), std::move(fmt),
+ std::move(fmap), std::move(source));
+ return ffi::Module(n);
+}
+
+} // namespace target
+} // namespace tvm
diff --git a/src/target/hexagon/hexagon_fallback_module.h b/src/target/hexagon/hexagon_fallback_module.h
new file mode 100644
index 0000000000..3a42ac5b8b
--- /dev/null
+++ b/src/target/hexagon/hexagon_fallback_module.h
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file hexagon_fallback_module.h
+ * \brief Codegen-facing Hexagon module factory.
+ *
+ * `HexagonModuleCreateWithFallback` is the ONLY entry point codegen uses
+ * to construct a Hexagon `ffi::Module`. It tries the runtime-registered
+ * factory "ffi.Module.create.hexagon" via the FFI registry; on miss it
+ * constructs a `HexagonFallbackModuleNode` directly. The fallback exists
+ * so that codegen can succeed on a build where the Hexagon runtime is
+ * not linked. This setup is helpful for cross compilation where we
+ * compile on one env and run on another.
+ */
+#ifndef TVM_TARGET_HEXAGON_HEXAGON_FALLBACK_MODULE_H_
+#define TVM_TARGET_HEXAGON_HEXAGON_FALLBACK_MODULE_H_
+
+#include <tvm/ffi/container/map.h>
+#include <tvm/ffi/container/variant.h>
+#include <tvm/ffi/extra/module.h>
+#include <tvm/ffi/function.h>
+
+#include <utility>
+
+#include "../../runtime/metadata.h"
+#include "../../support/env.h"
+
+namespace tvm {
+namespace target {
+
+/*! \brief Source-map type for Hexagon — the only backend with Variant. */
+using HexagonSourceMap = ffi::Map<ffi::String, ffi::Variant<ffi::String, ffi::Bytes>>;
+
+/*!
+ * \brief Construct a `HexagonFallbackModuleNode` directly (no FFI registry
+ * round-trip). Used by `HexagonModuleCreateWithFallback` and by
+ * tests that explicitly want a fallback instance.
+ */
+ffi::Module HexagonFallbackModuleCreate(ffi::Bytes code, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ HexagonSourceMap source);
+
+/*!
+ * \brief Codegen-time Hexagon module factory. Tries the FFI-registered
+ * "ffi.Module.create.hexagon"; on miss or when forced via env var,
+ * falls back to `HexagonFallbackModuleCreate`.
+ *
+ * - Registry hit (USE_HEXAGON=ON build) → real `HexagonModuleNode`
+ * returned.
+ * - Registry miss (USE_HEXAGON=OFF build) → `HexagonFallbackModuleNode`
+ * returned.
+ * - `TVM_COMPILE_FORCE_FALLBACK` env var truthy → fallback regardless of
+ * registry state (used by per-backend fallback tests on a USE_X=ON CI
+ * box).
+ */
+inline ffi::Module HexagonModuleCreateWithFallback(
+ ffi::Bytes code, ffi::String fmt, ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ HexagonSourceMap source) {
+ if (tvm::support::GetEnv<bool>("TVM_COMPILE_FORCE_FALLBACK", false)) {
+ return HexagonFallbackModuleCreate(std::move(code), std::move(fmt), std::move(fmap),
+ std::move(source));
+ }
+ // Registry: "ffi.Module.create.hexagon" — real Hexagon runtime factory.
+ // Grep hint: grep -rn 'ffi.Module.create.hexagon' src/
+ auto fcreate = ffi::Function::GetGlobal("ffi.Module.create.hexagon");
+ if (fcreate.has_value()) {
+ return (*fcreate)(code, fmt, fmap, source).cast<ffi::Module>();
+ }
+ return HexagonFallbackModuleCreate(std::move(code), std::move(fmt), std::move(fmap),
+ std::move(source));
+}
+
+} // namespace target
+} // namespace tvm
+#endif // TVM_TARGET_HEXAGON_HEXAGON_FALLBACK_MODULE_H_
diff --git a/src/target/llvm/codegen_hexagon.cc b/src/target/hexagon/llvm/codegen_hexagon.cc
similarity index 95%
rename from src/target/llvm/codegen_hexagon.cc
rename to src/target/hexagon/llvm/codegen_hexagon.cc
index 28fa7f1edd..a7dbda398d 100644
--- a/src/target/llvm/codegen_hexagon.cc
+++ b/src/target/hexagon/llvm/codegen_hexagon.cc
@@ -55,10 +55,12 @@
#include <utility>
#include <vector>
-#include "../../runtime/hexagon/hexagon_module.h"
-#include "../build_common.h"
-#include "codegen_cpu.h"
-#include "llvm_instance.h"
+#include "../../../runtime/file_utils.h"
+#include "../../../runtime/metadata.h"
+#include "../../build_common.h"
+#include "../../llvm/codegen_cpu.h"
+#include "../../llvm/llvm_instance.h"
+#include "../hexagon_fallback_module.h"
namespace tvm {
namespace codegen {
@@ -568,7 +570,22 @@ ffi::Module BuildHexagon(IRModule mod, Target target) {
int rc = (*f)(so_name, o_names, extra_args).cast<int>();
TVM_FFI_ICHECK(rc == 0) << "Failed to link " << so_name;
- return HexagonModuleCreate(so_name, "so", ExtractFuncInfo(mod), asm_str, obj_str, ir_str, bc_str);
+ // The new design takes Bytes (not a filename); slurp the linked .so into
+ // memory. Receiver materializes the bytes back to a tempfile internally
+ // before dlopen.
+ std::string so_data;
+ runtime::LoadBinaryFromFile(so_name, &so_data);
+
+ // Pack auxiliaries into the unified source map. Variant lets text blobs
+ // (asm/ll) and binary blobs (obj/bc) coexist in their natural form.
+ target::HexagonSourceMap source;
+ source.Set("s", ffi::Variant<ffi::String, ffi::Bytes>(ffi::String(asm_str)));
+ source.Set("ll", ffi::Variant<ffi::String, ffi::Bytes>(ffi::String(ir_str)));
+ source.Set("obj", ffi::Variant<ffi::String, ffi::Bytes>(ffi::Bytes(obj_str)));
+ source.Set("bc", ffi::Variant<ffi::String, ffi::Bytes>(ffi::Bytes(bc_str)));
+
+ return target::HexagonModuleCreateWithFallback(ffi::Bytes(std::move(so_data)), "so",
+ ExtractFuncInfo(mod), std::move(source));
}
TVM_FFI_STATIC_INIT_BLOCK() {
diff --git a/src/target/llvm/intrin_rule_hexagon.cc b/src/target/hexagon/llvm/intrin_rule_hexagon.cc
similarity index 99%
rename from src/target/llvm/intrin_rule_hexagon.cc
rename to src/target/hexagon/llvm/intrin_rule_hexagon.cc
index e330dba4e1..cc5328b2b3 100644
--- a/src/target/llvm/intrin_rule_hexagon.cc
+++ b/src/target/hexagon/llvm/intrin_rule_hexagon.cc
@@ -23,7 +23,7 @@
#include <tvm/tirx/op.h>
#include <tvm/tirx/op_attr_types.h>
-#include "intrin_rule_llvm.h"
+#include "../../llvm/intrin_rule_llvm.h"
#define TVM_REGISTER_QHL_OP_FP16(INTRIN_FUNC, WRAPPER_FUNC, NUM_SIGN) \
std::string tvm_qhl_ahf_##INTRIN_FUNC = WRAPPER_FUNC; \
diff --git a/src/target/source/codegen_metal.cc b/src/target/metal/codegen_metal.cc
similarity index 95%
rename from src/target/source/codegen_metal.cc
rename to src/target/metal/codegen_metal.cc
index c4734c54bb..188e535805 100644
--- a/src/target/source/codegen_metal.cc
+++ b/src/target/metal/codegen_metal.cc
@@ -33,9 +33,9 @@
#include <unordered_map>
#include <utility>
-#include "../../runtime/metal/metal_module.h"
#include "../../runtime/thread_storage_scope.h"
#include "../build_common.h"
+#include "metal_fallback_module.h"
namespace tvm {
namespace codegen {
@@ -448,7 +448,11 @@ ffi::Module BuildMetal(IRModule mod, Target target) {
mod = tirx::transform::PointerValueTypeRewrite()(std::move(mod));
std::ostringstream source_maker;
- ffi::Map<ffi::String, ffi::String> smap;
+ // Per-kernel payload: kernel-name -> bytes (MSL source for fmt="metal" /
+ // compiled metallib bytes for fmt="metallib"). See
+ // tasks/...-tvm-unify-device-module... "Per-kernel smap shape" — uniform
+ // Map<String, Bytes> across all multi-shader backends.
+ ffi::Map<ffi::String, ffi::Bytes> smap;
const auto fmetal_compile = tvm::ffi::Function::GetGlobal("tvm_callback_metal_compile");
std::string fmt = fmetal_compile ? "metallib" : "metal";
@@ -473,10 +477,15 @@ ffi::Module BuildMetal(IRModule mod, Target target) {
if (fmetal_compile) {
fsource = (*fmetal_compile)(fsource, target).cast<std::string>();
}
- smap.Set(func_name, fsource);
+ smap.Set(func_name, ffi::Bytes(std::move(fsource)));
}
- return MetalModuleCreate(smap, ExtractFuncInfo(mod), fmt, source_maker.str());
+ // The aggregated MSL source dump is preserved in the in-memory source
+ // map keyed by "metal" — only used by InspectSource and never serialized.
+ ffi::Map<ffi::String, ffi::String> source;
+ source.Set("metal", source_maker.str());
+ return target::MetalModuleCreateWithFallback(std::move(smap), ffi::String(fmt),
+ ExtractFuncInfo(mod), std::move(source));
}
TVM_FFI_STATIC_INIT_BLOCK() {
diff --git a/src/target/source/codegen_metal.h b/src/target/metal/codegen_metal.h
similarity index 94%
rename from src/target/source/codegen_metal.h
rename to src/target/metal/codegen_metal.h
index 8122ede542..77884660a1 100644
--- a/src/target/source/codegen_metal.h
+++ b/src/target/metal/codegen_metal.h
@@ -21,15 +21,15 @@
* \file codegen_metal.h
* \brief Generate Metal device code.
*/
-#ifndef TVM_TARGET_SOURCE_CODEGEN_METAL_H_
-#define TVM_TARGET_SOURCE_CODEGEN_METAL_H_
+#ifndef TVM_TARGET_METAL_CODEGEN_METAL_H_
+#define TVM_TARGET_METAL_CODEGEN_METAL_H_
#include <tvm/target/codegen.h>
#include <string>
#include <unordered_map>
-#include "codegen_c.h"
+#include "../source/codegen_c.h"
namespace tvm {
namespace codegen {
@@ -69,4 +69,4 @@ class CodeGenMetal final : public CodeGenC {
} // namespace codegen
} // namespace tvm
-#endif // TVM_TARGET_SOURCE_CODEGEN_METAL_H_
+#endif // TVM_TARGET_METAL_CODEGEN_METAL_H_
diff --git a/src/target/source/intrin_rule_metal.cc b/src/target/metal/intrin_rule_metal.cc
similarity index 100%
rename from src/target/source/intrin_rule_metal.cc
rename to src/target/metal/intrin_rule_metal.cc
diff --git a/src/target/metal/metal_fallback_module.cc b/src/target/metal/metal_fallback_module.cc
new file mode 100644
index 0000000000..0954c307b4
--- /dev/null
+++ b/src/target/metal/metal_fallback_module.cc
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file metal_fallback_module.cc
+ * \brief MetalFallbackModuleNode — codegen-time placeholder used when the
+ * Metal runtime is not linked. Mirrors `MetalModuleNode`'s save/load
+ * format byte-for-byte; see one-way comment in `SaveToBytes` below.
+ * Always compiled (independent of USE_METAL); never registered as an
+ * FFI factory or loader. Plain C++ — no Metal/Cocoa types.
+ */
+#include "metal_fallback_module.h"
+
+#include <tvm/ffi/extra/module.h>
+#include <tvm/ffi/function.h>
+
+#include <string>
+#include <utility>
+
+#include "../../support/bytes_io.h"
+
+namespace tvm {
+namespace target {
+
+class MetalFallbackModuleNode : public ffi::ModuleObj {
+ public:
+ MetalFallbackModuleNode(ffi::Map<ffi::String, ffi::Bytes> smap, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source)
+ : smap_(std::move(smap)),
+ fmt_(std::move(fmt)),
+ fmap_(std::move(fmap)),
+ source_(std::move(source)) {}
+
+ // Mirror the real module's kind so consumers cannot distinguish at the
+ // kind/api layer. Saved bytes load back as a real MetalModuleNode on a
+ // Metal-equipped (macOS, USE_METAL=ON) receiver.
+ const char* kind() const final { return "metal"; }
+
+ int GetPropertyMask() const final { return ffi::Module::kBinarySerializable; }
+
+ ffi::Optional<ffi::Function> GetFunction(const ffi::String& name) final {
+ TVM_FFI_THROW(RuntimeError)
+ << "Metal runtime is not linked into this build; cannot launch kernels. "
+ << "Re-link with USE_METAL=ON (macOS) or load this module in a "
+ << "Metal-equipped environment via tvm.runtime.load_module.";
+ TVM_FFI_UNREACHABLE();
+ }
+
+ ffi::Bytes SaveToBytes() const final {
+ // NOTE: serialization format MUST remain byte-identical to
+ // MetalModuleNode::SaveToBytes in src/runtime/metal/metal_module.mm
+ // (the source of truth). Both produce a kind="metal" artifact that
+ // the loader (ffi.Module.load_from_bytes.metal, registered only when
+ // USE_METAL=ON) deserializes. If the real impl's format changes,
+ // mirror the change here. The dependency is one-way: this file
+ // follows; metal_module.mm does not reference this file.
+ //
+ // 3 fields only — the source map is in-memory inspection material and
+ // is NEVER serialized (matches upstream behavior for all backends).
+ std::string buffer;
+ support::BytesOutStream stream(&buffer);
+ stream.Write(fmt_);
+ stream.Write(fmap_);
+ stream.Write(smap_);
+ return ffi::Bytes(std::move(buffer));
+ }
+
+ ffi::String InspectSource(const ffi::String& format) const final {
+ if (auto it = source_.find(format); it != source_.end()) {
+ return (*it).second;
+ }
+ if (format.empty()) {
+ // Default: aggregated MSL source dump (key "metal").
+ if (auto it = source_.find("metal"); it != source_.end()) {
+ return (*it).second;
+ }
+ // Fallback: concatenate all kernel sources from the smap (handle
+ // the fmt=="metal" text case).
+ if (fmt_ == "metal") {
+ std::string out;
+ for (const auto& kv : smap_) {
+ out.append(kv.second.data(), kv.second.size());
+ out.push_back('\n');
+ }
+ return ffi::String(std::move(out));
+ }
+ }
+ return ffi::String();
+ }
+
+ private:
+ // Per-kernel payload: kernel-name -> bytes. For fmt=="metal" each value
+ // is MSL source bytes; for fmt=="metallib" each value is a compiled
+ // metallib blob. Multi-shader uniform Map<String, Bytes> (see
+ // tasks/...-tvm-unify-device-module... "Per-kernel smap shape").
+ ffi::Map<ffi::String, ffi::Bytes> smap_;
+ // Format identifier ("metal" source / "metallib" compiled).
+ ffi::String fmt_;
+ // function information table.
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap_;
+ // In-memory source map for InspectSource — never serialized.
+ ffi::Map<ffi::String, ffi::String> source_;
+};
+
+ffi::Module MetalFallbackModuleCreate(ffi::Map<ffi::String, ffi::Bytes> smap, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source) {
+ auto n = ffi::make_object<MetalFallbackModuleNode>(std::move(smap), std::move(fmt),
+ std::move(fmap), std::move(source));
+ return ffi::Module(n);
+}
+
+} // namespace target
+} // namespace tvm
diff --git a/src/target/metal/metal_fallback_module.h b/src/target/metal/metal_fallback_module.h
new file mode 100644
index 0000000000..b30ebd9912
--- /dev/null
+++ b/src/target/metal/metal_fallback_module.h
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file metal_fallback_module.h
+ * \brief Codegen-facing Metal module factory.
+ *
+ * `MetalModuleCreateWithFallback` is the ONLY entry point codegen uses to
+ * construct a Metal `ffi::Module`. It tries the runtime-registered factory
+ * "ffi.Module.create.metal" via the FFI registry; on miss it constructs a
+ * `MetalFallbackModuleNode` directly. The fallback exists so that codegen
+ * can succeed on a build where the Metal runtime is not linked. This
+ * setup is helpful for cross compilation where we compile on one env
+ * and run on another.
+ */
+#ifndef TVM_TARGET_METAL_METAL_FALLBACK_MODULE_H_
+#define TVM_TARGET_METAL_METAL_FALLBACK_MODULE_H_
+
+#include <tvm/ffi/container/map.h>
+#include <tvm/ffi/extra/module.h>
+#include <tvm/ffi/function.h>
+
+#include <utility>
+
+#include "../../runtime/metadata.h"
+#include "../../support/env.h"
+
+namespace tvm {
+namespace target {
+
+/*!
+ * \brief Construct a `MetalFallbackModuleNode` directly (no FFI registry
+ * round-trip). Used by `MetalModuleCreateWithFallback` and by tests
+ * that explicitly want a fallback instance.
+ */
+ffi::Module MetalFallbackModuleCreate(ffi::Map<ffi::String, ffi::Bytes> smap, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source);
+
+/*!
+ * \brief Codegen-time Metal module factory. Tries the FFI-registered
+ * "ffi.Module.create.metal"; on miss or when forced via env var,
+ * falls back to `MetalFallbackModuleCreate`.
+ *
+ * - Registry hit (USE_METAL=ON build, macOS) → real `MetalModuleNode`
+ * returned.
+ * - Registry miss (USE_METAL=OFF build) → `MetalFallbackModuleNode`
+ * returned.
+ * - `TVM_COMPILE_FORCE_FALLBACK` env var truthy → fallback regardless of
+ * registry state (used by per-backend fallback tests on a USE_X=ON CI
+ * box).
+ */
+inline ffi::Module MetalModuleCreateWithFallback(ffi::Map<ffi::String, ffi::Bytes> smap,
+ ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source) {
+ if (tvm::support::GetEnv<bool>("TVM_COMPILE_FORCE_FALLBACK", false)) {
+ return MetalFallbackModuleCreate(std::move(smap), std::move(fmt), std::move(fmap),
+ std::move(source));
+ }
+ // Registry: "ffi.Module.create.metal" — real Metal runtime factory.
+ // Grep hint: grep -rn 'ffi.Module.create.metal' src/
+ auto fcreate = ffi::Function::GetGlobal("ffi.Module.create.metal");
+ if (fcreate.has_value()) {
+ return (*fcreate)(smap, fmt, fmap, source).cast<ffi::Module>();
+ }
+ return MetalFallbackModuleCreate(std::move(smap), std::move(fmt), std::move(fmap),
+ std::move(source));
+}
+
+} // namespace target
+} // namespace tvm
+#endif // TVM_TARGET_METAL_METAL_FALLBACK_MODULE_H_
diff --git a/src/target/source/codegen_opencl.cc b/src/target/opencl/codegen_opencl.cc
similarity index 98%
rename from src/target/source/codegen_opencl.cc
rename to src/target/opencl/codegen_opencl.cc
index b2f78c2dbd..52a1ce2e57 100644
--- a/src/target/source/codegen_opencl.cc
+++ b/src/target/opencl/codegen_opencl.cc
@@ -28,11 +28,10 @@
#include <string>
#include <vector>
-#include "../../runtime/opencl/opencl_module.h"
#include "../../runtime/texture.h"
#include "../../runtime/thread_storage_scope.h"
#include "../build_common.h"
-#include "../spirv/spirv_utils.h"
+#include "opencl_fallback_module.h"
namespace tvm {
namespace codegen {
@@ -682,14 +681,6 @@ void CodeGenOpenCL::SetTextureScope(
}
ffi::Module BuildOpenCL(IRModule mod, Target target) {
-#if TVM_ENABLE_SPIRV
- ffi::Optional<ffi::String> device = target->GetAttr<ffi::String>("device");
- if (device && device.value() == "spirv") {
- auto [smap, spirv_text] = LowerToSPIRV(mod, target);
- return runtime::OpenCLModuleCreate(smap, spirv_text, ExtractFuncInfo(mod));
- }
-#endif
-
bool output_ssa = false;
ffi::Map<GlobalVar, PrimFunc> functions;
@@ -720,7 +711,14 @@ ffi::Module BuildOpenCL(IRModule mod, Target target) {
code << fsource;
}
- return OpenCLModuleCreate(code.str(), "cl", ExtractFuncInfo(mod), code.str());
+ // The aggregated OpenCL C source dump is preserved in the in-memory
+ // source map keyed by "cl" — only used by InspectSource and never
+ // serialized.
+ std::string code_str = code.str();
+ ffi::Map<ffi::String, ffi::String> source;
+ source.Set("cl", code_str);
+ return target::OpenCLModuleCreateWithFallback(ffi::Bytes(code_str), ffi::String("cl"),
+ ExtractFuncInfo(mod), std::move(source));
}
TVM_FFI_STATIC_INIT_BLOCK() {
diff --git a/src/target/source/codegen_opencl.h b/src/target/opencl/codegen_opencl.h
similarity index 96%
rename from src/target/source/codegen_opencl.h
rename to src/target/opencl/codegen_opencl.h
index cd425f9f49..614aa60f47 100644
--- a/src/target/source/codegen_opencl.h
+++ b/src/target/opencl/codegen_opencl.h
@@ -21,15 +21,15 @@
* \file codegen_opencl.h
* \brief Generate OpenCL device code.
*/
-#ifndef TVM_TARGET_SOURCE_CODEGEN_OPENCL_H_
-#define TVM_TARGET_SOURCE_CODEGEN_OPENCL_H_
+#ifndef TVM_TARGET_OPENCL_CODEGEN_OPENCL_H_
+#define TVM_TARGET_OPENCL_CODEGEN_OPENCL_H_
#include <tvm/target/codegen.h>
#include <string>
#include <unordered_map>
-#include "codegen_c.h"
+#include "../source/codegen_c.h"
namespace tvm {
namespace codegen {
@@ -93,4 +93,4 @@ class CodeGenOpenCL final : public CodeGenC {
} // namespace codegen
} // namespace tvm
-#endif // TVM_TARGET_SOURCE_CODEGEN_OPENCL_H_
+#endif // TVM_TARGET_OPENCL_CODEGEN_OPENCL_H_
diff --git a/src/target/source/intrin_rule_opencl.cc b/src/target/opencl/intrin_rule_opencl.cc
similarity index 100%
rename from src/target/source/intrin_rule_opencl.cc
rename to src/target/opencl/intrin_rule_opencl.cc
diff --git a/src/target/opencl/opencl_fallback_module.cc b/src/target/opencl/opencl_fallback_module.cc
new file mode 100644
index 0000000000..454b254715
--- /dev/null
+++ b/src/target/opencl/opencl_fallback_module.cc
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file opencl_fallback_module.cc
+ * \brief OpenCLFallbackModuleNode — codegen-time placeholder used when
+ * the OpenCL runtime is not linked. Mirrors `OpenCLModuleNode`'s
+ * save/load format byte-for-byte; see one-way comment in
+ * `SaveToBytes` below. Always compiled (independent of
+ * USE_OPENCL); never registered as an FFI factory or loader.
+ * Plain C++ — no OpenCL-API types.
+ */
+#include "opencl_fallback_module.h"
+
+#include <tvm/ffi/extra/module.h>
+#include <tvm/ffi/function.h>
+
+#include <string>
+#include <utility>
+
+#include "../../support/bytes_io.h"
+
+namespace tvm {
+namespace target {
+
+class OpenCLFallbackModuleNode : public ffi::ModuleObj {
+ public:
+ OpenCLFallbackModuleNode(ffi::Bytes code, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source)
+ : code_(std::move(code)),
+ fmt_(std::move(fmt)),
+ fmap_(std::move(fmap)),
+ source_(std::move(source)) {}
+
+ // Mirror the real module's kind ("opencl") so consumers cannot
+ // distinguish at the kind/api layer. Saved bytes load back as a real
+ // OpenCLModuleNode on a USE_OPENCL=ON receiver.
+ const char* kind() const final { return "opencl"; }
+
+ int GetPropertyMask() const final { return ffi::Module::kBinarySerializable; }
+
+ ffi::Optional<ffi::Function> GetFunction(const ffi::String& name) final {
+ TVM_FFI_THROW(RuntimeError)
+ << "OpenCL runtime is not linked into this build; cannot launch kernels. "
+ << "Re-link with USE_OPENCL=ON or load this module in an OpenCL-equipped "
+ << "environment via tvm.runtime.load_module.";
+ TVM_FFI_UNREACHABLE();
+ }
+
+ ffi::Bytes SaveToBytes() const final {
+ // NOTE: serialization format MUST remain byte-identical to
+ // OpenCLModuleNode::SaveToBytes in src/runtime/opencl/opencl_module.cc
+ // (the source of truth). Both produce a kind="opencl" artifact
+ // that the loader (ffi.Module.load_from_bytes.opencl, registered
+ // only when USE_OPENCL=ON) deserializes. If the real impl's
+ // format changes, mirror the change here. The dependency is
+ // one-way: this file follows; opencl_module.cc does not reference
+ // this file.
+ //
+ // 3 fields only — the source map is in-memory inspection material
+ // and is NEVER serialized (matches upstream behavior for all
+ // backends).
+ std::string buffer;
+ support::BytesOutStream stream(&buffer);
+ stream.Write(fmt_);
+ stream.Write(fmap_);
+ stream.Write(code_);
+ return ffi::Bytes(std::move(buffer));
+ }
+
+ ffi::String InspectSource(const ffi::String& format) const final {
+ if (auto it = source_.find(format); it != source_.end()) {
+ return (*it).second;
+ }
+ if (format.empty()) {
+ // Default: aggregated OpenCL C source dump (key "cl").
+ if (auto it = source_.find("cl"); it != source_.end()) {
+ return (*it).second;
+ }
+ // Fallback: if fmt=="cl" the code is itself the OpenCL source.
+ if (fmt_ == "cl") {
+ return ffi::String(code_.data(), code_.size());
+ }
+ }
+ return ffi::String();
+ }
+
+ private:
+ // Single-binary payload: for fmt=="cl" the bytes are the OpenCL C
+ // source; for fmt=="xclbin"/"awsxclbin"/"aocx" the bytes are a
+ // pre-compiled OpenCL binary. Single-path uniform Bytes (vs
+ // multi-shader Map<String, Bytes> on Vulkan/Metal/WebGPU).
+ ffi::Bytes code_;
+ // Format identifier ("cl" / "xclbin" / "awsxclbin" / "aocx").
+ ffi::String fmt_;
+ // function information table.
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap_;
+ // In-memory source map for InspectSource — never serialized.
+ ffi::Map<ffi::String, ffi::String> source_;
+};
+
+ffi::Module OpenCLFallbackModuleCreate(ffi::Bytes code, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source) {
+ auto n = ffi::make_object<OpenCLFallbackModuleNode>(std::move(code), std::move(fmt),
+ std::move(fmap), std::move(source));
+ return ffi::Module(n);
+}
+
+} // namespace target
+} // namespace tvm
diff --git a/src/target/opencl/opencl_fallback_module.h b/src/target/opencl/opencl_fallback_module.h
new file mode 100644
index 0000000000..ef401c7547
--- /dev/null
+++ b/src/target/opencl/opencl_fallback_module.h
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file opencl_fallback_module.h
+ * \brief Codegen-facing OpenCL module factory.
+ *
+ * `OpenCLModuleCreateWithFallback` is the ONLY entry point codegen uses
+ * to construct an OpenCL `ffi::Module`. It tries the runtime-registered
+ * factory "ffi.Module.create.opencl" via the FFI registry; on miss it
+ * constructs an `OpenCLFallbackModuleNode` directly. The fallback exists
+ * so that codegen can succeed on a build where the OpenCL runtime is
+ * not linked. This setup is helpful for cross compilation where we
+ * compile on one env and run on another.
+ */
+#ifndef TVM_TARGET_OPENCL_OPENCL_FALLBACK_MODULE_H_
+#define TVM_TARGET_OPENCL_OPENCL_FALLBACK_MODULE_H_
+
+#include <tvm/ffi/container/map.h>
+#include <tvm/ffi/extra/module.h>
+#include <tvm/ffi/function.h>
+
+#include <utility>
+
+#include "../../runtime/metadata.h"
+#include "../../support/env.h"
+
+namespace tvm {
+namespace target {
+
+/*!
+ * \brief Construct an `OpenCLFallbackModuleNode` directly (no FFI
+ * registry round-trip). Used by `OpenCLModuleCreateWithFallback`
+ * and by tests that explicitly want a fallback instance.
+ */
+ffi::Module OpenCLFallbackModuleCreate(ffi::Bytes code, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source);
+
+/*!
+ * \brief Codegen-time OpenCL module factory. Tries the FFI-registered
+ * "ffi.Module.create.opencl"; on miss or when forced via env
+ * var, falls back to `OpenCLFallbackModuleCreate`.
+ *
+ * - Registry hit (USE_OPENCL=ON build) → real OpenCL module returned.
+ * - Registry miss (USE_OPENCL=OFF build) → `OpenCLFallbackModuleNode`
+ * returned.
+ * - `TVM_COMPILE_FORCE_FALLBACK` env var truthy → fallback regardless
+ * of registry state (used by per-backend fallback tests on a
+ * USE_X=ON CI box).
+ */
+inline ffi::Module OpenCLModuleCreateWithFallback(ffi::Bytes code, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source) {
+ if (tvm::support::GetEnv<bool>("TVM_COMPILE_FORCE_FALLBACK", false)) {
+ return OpenCLFallbackModuleCreate(std::move(code), std::move(fmt), std::move(fmap),
+ std::move(source));
+ }
+ // Registry: "ffi.Module.create.opencl" — real OpenCL runtime factory.
+ // Grep hint: grep -rn 'ffi.Module.create.opencl' src/
+ auto fcreate = ffi::Function::GetGlobal("ffi.Module.create.opencl");
+ if (fcreate.has_value()) {
+ return (*fcreate)(code, fmt, fmap, source).cast<ffi::Module>();
+ }
+ return OpenCLFallbackModuleCreate(std::move(code), std::move(fmt), std::move(fmap),
+ std::move(source));
+}
+
+} // namespace target
+} // namespace tvm
+#endif // TVM_TARGET_OPENCL_OPENCL_FALLBACK_MODULE_H_
diff --git a/src/target/opt/build_cuda_off.cc b/src/target/opt/build_cuda_off.cc
deleted file mode 100644
index e9e4351c89..0000000000
--- a/src/target/opt/build_cuda_off.cc
+++ /dev/null
@@ -1,24 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-/*!
- * Optional module when build CUDA is switched to off.
- * CUDAModuleCreate is now an inline registry-lookup wrapper in cuda_module.h,
- * so no out-of-line stub is needed here.
- */
diff --git a/src/target/opt/build_cuda_on.cc b/src/target/opt/build_cuda_on.cc
deleted file mode 100644
index 4e312e93a4..0000000000
--- a/src/target/opt/build_cuda_on.cc
+++ /dev/null
@@ -1,112 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-/*!
- * Build CUDA modules from source.
- * requires CUDA to be available.
- *
- * \file build_cuda.cc
- */
-#if defined(__linux__)
-#include <sys/stat.h>
-#include <tvm/ffi/reflection/registry.h>
-#endif
-#include <cuda_runtime.h>
-
-#include <cstdlib>
-
-#include "../../runtime/cuda/cuda_common.h"
-#include "../../runtime/cuda/cuda_module.h"
-#include "../build_common.h"
-#include "../source/codegen_cuda.h"
-
-namespace tvm {
-namespace codegen {
-
-// Note: CUDA include path finding and NVRTC compilation are now handled
-// in Python for better maintainability and to leverage cuda-python bindings.
-// The C++ NVRTC code has been removed as part of the Python-first
-// compilation strategy.
-
-ffi::Module BuildCUDA(IRModule mod, Target target) {
- bool output_ssa = false;
- CodeGenCUDA cg;
- cg.Init(output_ssa);
-
- ffi::Map<GlobalVar, PrimFunc> functions;
- for (auto [gvar, base_func] : mod->functions) {
- TVM_FFI_ICHECK(base_func->IsInstance<PrimFuncNode>()) << "CodeGenCUDA: Can only take PrimFunc";
- auto prim_func = Downcast<PrimFunc>(base_func);
- auto calling_conv =
- prim_func->GetAttr<Integer>(tvm::attr::kCallingConv, Integer(tvm::CallingConv::kDefault));
- TVM_FFI_ICHECK(calling_conv == CallingConv::kDeviceKernelLaunch ||
- calling_conv == CallingConv::kDefault)
- << "CodeGenCUDA: expect calling_conv equals CallingConv::kDeviceKernelLaunch or "
- "CallingConv::kDefault";
- functions.Set(gvar, prim_func);
- }
-
- for (auto [gvar, prim_func] : functions) {
- cg.DeclareFunction(gvar, prim_func);
- }
- for (auto [gvar, prim_func] : functions) {
- cg.AddFunction(gvar, prim_func);
- }
-
- std::string code = cg.Finish();
-
- if (auto f = ffi::Function::GetGlobal("tvm_callback_cuda_postproc")) {
- code = (*f)(code, target).cast<std::string>();
- }
- std::string fmt = "ptx";
- std::string compiled;
-
- // Always use Python compilation callback (nvcc or nvrtc)
- // The C++ NVRTC fallback has been removed in favor of Python-first approach
- auto f_compile = ffi::Function::GetGlobal("tvm_callback_cuda_compile");
- TVM_FFI_ICHECK(f_compile != nullptr)
- << "tvm_callback_cuda_compile not found. "
- << "Please ensure TVM Python runtime is properly initialized.\n"
- << "The Python callback (tvm.contrib.nvcc.tvm_callback_cuda_compile) is required "
- << "for CUDA compilation. The C++ NVRTC fallback has been removed.\n"
- << "Make sure to import tvm.contrib.nvcc in your Python code.";
-
- // Enter target scope for compilation
- auto f_enter = ffi::Function::GetGlobal("target.TargetEnterScope");
- (*f_enter)(target);
-
- // Compile CUDA code via Python callback
- compiled = (*f_compile)(code, target).cast<std::string>();
- // Dirty matching to check PTX vs cubin.
- // TODO(tqchen) more reliable checks
- if (compiled[0] != '/') fmt = "cubin";
- // Exit target scope
- auto f_exit = ffi::Function::GetGlobal("target.TargetExitScope");
- (*f_exit)(target);
-
- return CUDAModuleCreate(compiled, fmt, ExtractFuncInfo(mod), code);
-}
-
-TVM_FFI_STATIC_INIT_BLOCK() {
- namespace refl = tvm::ffi::reflection;
- refl::GlobalDef().def("target.build.cuda", BuildCUDA);
-}
-TVM_REGISTER_PASS_CONFIG_OPTION("cuda.kernels_output_dir", ffi::String);
-} // namespace codegen
-} // namespace tvm
diff --git a/src/target/opt/build_hexagon_off.cc b/src/target/opt/build_hexagon_off.cc
deleted file mode 100644
index 08450cf171..0000000000
--- a/src/target/opt/build_hexagon_off.cc
+++ /dev/null
@@ -1,49 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-/*!
- * Optional module when Hexagon runtime is switched to off.
- * When ffi.Module.create.hexagon is not registered, HexagonModuleCreate (the inline
- * wrapper) raises a clear RuntimeError. Fall back to a DeviceSourceModule for
- * compilation-only (source inspection) workflows instead.
- */
-#include "../../runtime/hexagon/hexagon_module.h"
-#include "../source/codegen_source_base.h"
-
-namespace tvm {
-namespace runtime {
-
-// Register a fallback creator so that compiler-side code that calls
-// HexagonModuleCreate() when USE_HEXAGON=OFF still gets a usable
-// DeviceSourceModule (for source inspection / serialisation) rather than a
-// registry-not-found error.
-TVM_FFI_STATIC_INIT_BLOCK() {
- namespace refl = tvm::ffi::reflection;
- refl::GlobalDef().def(
- "ffi.Module.create.hexagon",
- [](ffi::String data, ffi::String fmt, ffi::Map<ffi::String, FunctionInfo> fmap,
- ffi::String /*asm_str*/, ffi::String /*obj_str*/, ffi::String /*ir_str*/,
- ffi::String /*bc_str*/) -> ffi::Module {
- LOG(WARNING) << "Hexagon runtime is not enabled, returning a source module...";
- return codegen::DeviceSourceModuleCreate(std::string(data), std::string(fmt), fmap, "hex");
- });
-}
-
-} // namespace runtime
-} // namespace tvm
diff --git a/src/target/opt/build_metal_off.cc b/src/target/opt/build_metal_off.cc
deleted file mode 100644
index fae5d511c6..0000000000
--- a/src/target/opt/build_metal_off.cc
+++ /dev/null
@@ -1,24 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-/*!
- * Optional module when build metal is switched to off.
- * MetalModuleCreate is now an inline registry-lookup wrapper in metal_module.h,
- * so no out-of-line stub is needed here.
- */
diff --git a/src/target/opt/build_opencl_off.cc b/src/target/opt/build_opencl_off.cc
deleted file mode 100644
index e30725f442..0000000000
--- a/src/target/opt/build_opencl_off.cc
+++ /dev/null
@@ -1,53 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-/*!
- * Optional module when build opencl is switched to off.
- * Register fallback creators so that compiler-side code (codegen_opencl.cc)
- * that calls OpenCLModuleCreate() when USE_OPENCL=OFF still gets a usable
- * DeviceSourceModule for source inspection / serialisation workflows.
- */
-#include <tvm/ffi/reflection/registry.h>
-
-#include "../../runtime/metadata.h"
-#include "../source/codegen_source_base.h"
-
-namespace tvm {
-namespace runtime {
-
-TVM_FFI_STATIC_INIT_BLOCK() {
- namespace refl = tvm::ffi::reflection;
- refl::GlobalDef()
- .def("ffi.Module.create.opencl",
- [](ffi::String data, ffi::String fmt, ffi::Map<ffi::String, FunctionInfo> fmap,
- ffi::String /*source*/) -> ffi::Module {
- return codegen::DeviceSourceModuleCreate(std::string(data), std::string(fmt), fmap,
- "opencl");
- })
- .def("ffi.Module.create.opencl.spirv",
- [](ffi::Map<ffi::String, ffi::Bytes> /*shader_bytes*/, ffi::String /*spirv_text*/,
- ffi::Map<ffi::String, FunctionInfo> /*fmap*/) -> ffi::Module {
- TVM_FFI_THROW(InternalError)
- << "OpenCLModuleCreate (SPIRV) is called but OpenCL is not enabled.";
- TVM_FFI_UNREACHABLE();
- });
-}
-
-} // namespace runtime
-} // namespace tvm
diff --git a/src/target/opt/build_rocm_off.cc b/src/target/opt/build_rocm_off.cc
deleted file mode 100644
index 634c8252c8..0000000000
--- a/src/target/opt/build_rocm_off.cc
+++ /dev/null
@@ -1,24 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-/*!
- * Optional module when build rocm is switched to off.
- * ROCMModuleCreate is now an inline registry-lookup wrapper in rocm_module.h,
- * so no out-of-line stub is needed here.
- */
diff --git a/src/target/llvm/codegen_amdgpu.cc b/src/target/rocm/llvm/codegen_amdgpu.cc
similarity index 96%
rename from src/target/llvm/codegen_amdgpu.cc
rename to src/target/rocm/llvm/codegen_amdgpu.cc
index a524dcce6d..ef31805be7 100644
--- a/src/target/llvm/codegen_amdgpu.cc
+++ b/src/target/rocm/llvm/codegen_amdgpu.cc
@@ -48,10 +48,11 @@
#include <tvm/runtime/base.h>
#include <tvm/runtime/device_api.h>
-#include "../../runtime/rocm/rocm_module.h"
-#include "../build_common.h"
-#include "codegen_llvm.h"
-#include "llvm_instance.h"
+#include "../../../runtime/metadata.h"
+#include "../../build_common.h"
+#include "../../llvm/codegen_llvm.h"
+#include "../../llvm/llvm_instance.h"
+#include "../rocm_fallback_module.h"
namespace tvm {
namespace codegen {
@@ -312,7 +313,11 @@ ffi::Module BuildAMDGPU(IRModule mod, Target target) {
std::string hsaco = (*flink)(&arr).cast<std::string>();
std::string ll(data_ll.begin(), data_ll.end());
- return ROCMModuleCreate(hsaco, "hsaco", ExtractFuncInfo(mod), ll, assembly);
+ ffi::Map<ffi::String, ffi::String> source;
+ source.Set("hip", ffi::String(ll));
+ source.Set("asm", ffi::String(assembly));
+ return target::ROCmModuleCreateWithFallback(ffi::Bytes(std::move(hsaco)), "hsaco",
+ ExtractFuncInfo(mod), std::move(source));
}
TVM_FFI_STATIC_INIT_BLOCK() {
diff --git a/src/target/llvm/intrin_rule_rocm.cc b/src/target/rocm/llvm/intrin_rule_rocm.cc
similarity index 99%
rename from src/target/llvm/intrin_rule_rocm.cc
rename to src/target/rocm/llvm/intrin_rule_rocm.cc
index 4d542c1299..8849270b03 100644
--- a/src/target/llvm/intrin_rule_rocm.cc
+++ b/src/target/rocm/llvm/intrin_rule_rocm.cc
@@ -31,8 +31,8 @@
#include <sstream>
-#include "../intrin_rule.h"
-#include "intrin_rule_llvm.h"
+#include "../../intrin_rule.h"
+#include "../../llvm/intrin_rule_llvm.h"
namespace tvm {
namespace codegen {
diff --git a/src/target/rocm/rocm_fallback_module.cc b/src/target/rocm/rocm_fallback_module.cc
new file mode 100644
index 0000000000..4492be30bb
--- /dev/null
+++ b/src/target/rocm/rocm_fallback_module.cc
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file rocm_fallback_module.cc
+ * \brief ROCmFallbackModuleNode — codegen-time placeholder used when the ROCm
+ * runtime is not linked. Mirrors `ROCMModuleNode`'s save/load format
+ * byte-for-byte; see one-way comment in `SaveToBytes` below.
+ * Always compiled (independent of USE_ROCM); never registered as an
+ * FFI factory or loader.
+ */
+#include "rocm_fallback_module.h"
+
+#include <tvm/ffi/extra/module.h>
+#include <tvm/ffi/function.h>
+
+#include <string>
+#include <utility>
+
+#include "../../support/bytes_io.h"
+
+namespace tvm {
+namespace target {
+
+class ROCmFallbackModuleNode : public ffi::ModuleObj {
+ public:
+ ROCmFallbackModuleNode(ffi::Bytes code, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source)
+ : code_(std::move(code)),
+ fmt_(std::move(fmt)),
+ fmap_(std::move(fmap)),
+ source_(std::move(source)) {}
+
+ // Mirror the real module's kind so consumers cannot distinguish at the
+ // kind/api layer. Saved bytes load back as a real ROCMModuleNode on a
+ // ROCm-equipped receiver.
+ const char* kind() const final { return "hip"; }
+
+ int GetPropertyMask() const final { return ffi::Module::kBinarySerializable; }
+
+ ffi::Optional<ffi::Function> GetFunction(const ffi::String& name) final {
+ TVM_FFI_THROW(RuntimeError)
+ << "ROCm runtime is not linked into this build; cannot launch kernels. "
+ << "Re-link with USE_ROCM=ON or load this module in a ROCm-equipped "
+ << "environment via tvm.runtime.load_module.";
+ TVM_FFI_UNREACHABLE();
+ }
+
+ ffi::Bytes SaveToBytes() const final {
+ // NOTE: serialization format MUST remain byte-identical to
+ // ROCMModuleNode::SaveToBytes in src/runtime/rocm/rocm_module.cc (the
+ // source of truth). Both produce a kind="hip" artifact that the loader
+ // (ffi.Module.load_from_bytes.hsaco, registered only when USE_ROCM=ON)
+ // deserializes. If the real impl's format changes, mirror the change
+ // here. The dependency is one-way: this file follows; rocm_module.cc
+ // does not reference this file.
+ std::string buffer;
+ support::BytesOutStream stream(&buffer);
+ stream.Write(fmt_);
+ stream.Write(fmap_);
+ stream.Write(code_);
+ return ffi::Bytes(std::move(buffer));
+ }
+
+ ffi::String InspectSource(const ffi::String& format) const final {
+ if (format == fmt_) {
+ return ffi::String(code_.data(), code_.size());
+ }
+ if (auto it = source_.find(format); it != source_.end()) {
+ return (*it).second;
+ }
+ if (format.empty() || format == "llvm") {
+ // Backward-compat: legacy ROCMModuleNode returned `hip_source_` for both
+ // empty-format and "llvm". Map both to source["hip"] (the LLVM IR text
+ // emitted by the AMDGPU backend; populated by codegen_amdgpu.cc).
+ if (auto it = source_.find("hip"); it != source_.end()) {
+ return (*it).second;
+ }
+ }
+ if (format == "asm") {
+ if (auto it = source_.find("asm"); it != source_.end()) {
+ return (*it).second;
+ }
+ }
+ return ffi::String();
+ }
+
+ private:
+ // Whatever the codegen produced (compiled hsaco bytes — ROCm has no
+ // source-JIT path, so fmt is always "hsaco").
+ ffi::Bytes code_;
+ // Format identifier ("hsaco").
+ ffi::String fmt_;
+ // function information table.
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap_;
+ // In-memory source map for InspectSource — never serialized.
+ ffi::Map<ffi::String, ffi::String> source_;
+};
+
+ffi::Module ROCmFallbackModuleCreate(ffi::Bytes code, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source) {
+ auto n = ffi::make_object<ROCmFallbackModuleNode>(std::move(code), std::move(fmt),
+ std::move(fmap), std::move(source));
+ return ffi::Module(n);
+}
+
+} // namespace target
+} // namespace tvm
diff --git a/src/target/rocm/rocm_fallback_module.h b/src/target/rocm/rocm_fallback_module.h
new file mode 100644
index 0000000000..e9ef4cc073
--- /dev/null
+++ b/src/target/rocm/rocm_fallback_module.h
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file rocm_fallback_module.h
+ * \brief Codegen-facing ROCm module factory.
+ *
+ * `ROCmModuleCreateWithFallback` is the ONLY entry point codegen uses to
+ * construct a ROCm `ffi::Module`. It tries the runtime-registered factory
+ * "ffi.Module.create.rocm" via the FFI registry; on miss it constructs a
+ * `ROCmFallbackModuleNode` directly. The fallback exists so that codegen
+ * can succeed on a build where the ROCm runtime is not linked. This
+ * setup is helpful for cross compilation where we compile on one env
+ * and run on another.
+ */
+#ifndef TVM_TARGET_ROCM_ROCM_FALLBACK_MODULE_H_
+#define TVM_TARGET_ROCM_ROCM_FALLBACK_MODULE_H_
+
+#include <tvm/ffi/container/map.h>
+#include <tvm/ffi/extra/module.h>
+#include <tvm/ffi/function.h>
+
+#include <utility>
+
+#include "../../runtime/metadata.h"
+#include "../../support/env.h"
+
+namespace tvm {
+namespace target {
+
+/*!
+ * \brief Construct a `ROCmFallbackModuleNode` directly (no FFI registry
+ * round-trip). Used by `ROCmModuleCreateWithFallback` and by tests
+ * that explicitly want a fallback instance.
+ */
+ffi::Module ROCmFallbackModuleCreate(ffi::Bytes code, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source);
+
+/*!
+ * \brief Codegen-time ROCm module factory. Tries the FFI-registered
+ * "ffi.Module.create.rocm"; on miss or when forced via env var,
+ * falls back to `ROCmFallbackModuleCreate`.
+ *
+ * - Registry hit (USE_ROCM=ON build) → real `ROCMModuleNode` returned.
+ * - Registry miss (USE_ROCM=OFF build) → `ROCmFallbackModuleNode` returned.
+ * - `TVM_COMPILE_FORCE_FALLBACK` env var truthy → fallback regardless of
+ * registry state (used by per-backend fallback tests on a USE_X=ON CI
+ * box).
+ */
+inline ffi::Module ROCmModuleCreateWithFallback(ffi::Bytes code, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source) {
+ if (tvm::support::GetEnv<bool>("TVM_COMPILE_FORCE_FALLBACK", false)) {
+ return ROCmFallbackModuleCreate(std::move(code), std::move(fmt), std::move(fmap),
+ std::move(source));
+ }
+ // Registry: "ffi.Module.create.rocm" — real ROCm runtime factory.
+ // Grep hint: grep -rn 'ffi.Module.create.rocm' src/
+ auto fcreate = ffi::Function::GetGlobal("ffi.Module.create.rocm");
+ if (fcreate.has_value()) {
+ return (*fcreate)(code, fmt, fmap, source).cast<ffi::Module>();
+ }
+ return ROCmFallbackModuleCreate(std::move(code), std::move(fmt), std::move(fmap),
+ std::move(source));
+}
+
+} // namespace target
+} // namespace tvm
+#endif // TVM_TARGET_ROCM_ROCM_FALLBACK_MODULE_H_
diff --git a/src/target/spirv/build_vulkan.cc b/src/target/vulkan/build_vulkan.cc
similarity index 55%
rename from src/target/spirv/build_vulkan.cc
rename to src/target/vulkan/build_vulkan.cc
index f71b7ef8d6..ffadf5a240 100644
--- a/src/target/spirv/build_vulkan.cc
+++ b/src/target/vulkan/build_vulkan.cc
@@ -24,17 +24,37 @@
#include <tvm/ffi/reflection/registry.h>
-#include "../../runtime/spirv/spirv_shader.h"
-#include "../../runtime/vulkan/vulkan_module.h"
+#include <string>
+#include <utility>
+
+#include "../../runtime/vulkan/spirv_shader.h"
+#include "../../support/bytes_io.h"
#include "../build_common.h"
#include "spirv_utils.h"
+#include "vulkan_fallback_module.h"
namespace tvm {
namespace codegen {
ffi::Module BuildSPIRV(IRModule mod, Target target) {
auto [smap, spirv_text] = LowerToSPIRV(mod, target);
- return runtime::VulkanModuleCreate(smap, ExtractFuncInfo(mod), spirv_text);
+ // Serialize each SPIRVShader to ffi::Bytes for the unified per-kernel
+ // smap shape. Each value is a self-packed SPIRVShader (flag + data
+ // vector); the Vulkan runtime (USE_VULKAN=ON) deserializes via the
+ // inverse helper in src/runtime/vulkan/vulkan_module.cc.
+ ffi::Map<ffi::String, ffi::Bytes> shader_bytes;
+ for (auto& kv : smap) {
+ std::string buf;
+ support::BytesOutStream strm(&buf);
+ strm.Write(kv.second);
+ shader_bytes.Set(kv.first, ffi::Bytes(std::move(buf)));
+ }
+ // The aggregated SPIR-V text dump is preserved in the in-memory source
+ // map keyed by "spv" — only used by InspectSource and never serialized.
+ ffi::Map<ffi::String, ffi::String> source;
+ source.Set("spv", std::move(spirv_text));
+ return target::VulkanModuleCreateWithFallback(std::move(shader_bytes), ffi::String("vulkan"),
+ ExtractFuncInfo(mod), std::move(source));
}
TVM_FFI_STATIC_INIT_BLOCK() {
diff --git a/src/target/spirv/codegen_spirv.cc b/src/target/vulkan/codegen_spirv.cc
similarity index 100%
rename from src/target/spirv/codegen_spirv.cc
rename to src/target/vulkan/codegen_spirv.cc
diff --git a/src/target/spirv/codegen_spirv.h b/src/target/vulkan/codegen_spirv.h
similarity index 97%
rename from src/target/spirv/codegen_spirv.h
rename to src/target/vulkan/codegen_spirv.h
index a77ff82d3d..cea634d8ab 100644
--- a/src/target/spirv/codegen_spirv.h
+++ b/src/target/vulkan/codegen_spirv.h
@@ -21,8 +21,8 @@
* \file ir_builder.h
* \brief Utility for building SPIRV code block
*/
-#ifndef TVM_TARGET_SPIRV_CODEGEN_SPIRV_H_
-#define TVM_TARGET_SPIRV_CODEGEN_SPIRV_H_
+#ifndef TVM_TARGET_VULKAN_CODEGEN_SPIRV_H_
+#define TVM_TARGET_VULKAN_CODEGEN_SPIRV_H_
#include <tvm/arith/analyzer.h>
#include <tvm/target/target.h>
@@ -37,8 +37,8 @@
#include <utility>
#include <vector>
-#include "../../runtime/spirv/spirv_shader.h"
#include "../../runtime/thread_storage_scope.h"
+#include "../../runtime/vulkan/spirv_shader.h"
#include "ir_builder.h"
#include "spirv_support.h"
@@ -240,4 +240,4 @@ class CodeGenSPIRV : public ExprFunctor<spirv::Value(const PrimExpr&)>,
} // namespace codegen
} // namespace tvm
-#endif // TVM_TARGET_SPIRV_CODEGEN_SPIRV_H_
+#endif // TVM_TARGET_VULKAN_CODEGEN_SPIRV_H_
diff --git a/src/target/spirv/intrin_rule_spirv.cc b/src/target/vulkan/intrin_rule_spirv.cc
similarity index 100%
rename from src/target/spirv/intrin_rule_spirv.cc
rename to src/target/vulkan/intrin_rule_spirv.cc
diff --git a/src/target/spirv/ir_builder.cc b/src/target/vulkan/ir_builder.cc
similarity index 100%
rename from src/target/spirv/ir_builder.cc
rename to src/target/vulkan/ir_builder.cc
diff --git a/src/target/spirv/ir_builder.h b/src/target/vulkan/ir_builder.h
similarity index 99%
rename from src/target/spirv/ir_builder.h
rename to src/target/vulkan/ir_builder.h
index 2fab0ae088..3cca1b4cfe 100644
--- a/src/target/spirv/ir_builder.h
+++ b/src/target/vulkan/ir_builder.h
@@ -21,8 +21,8 @@
* \file ir_builder.h
* \brief Utility for building SPIRV code block
*/
-#ifndef TVM_TARGET_SPIRV_IR_BUILDER_H_
-#define TVM_TARGET_SPIRV_IR_BUILDER_H_
+#ifndef TVM_TARGET_VULKAN_IR_BUILDER_H_
+#define TVM_TARGET_VULKAN_IR_BUILDER_H_
#include <tvm/ffi/function.h>
#include <tvm/tirx/expr.h>
@@ -768,4 +768,4 @@ class IRBuilder {
} // namespace codegen
} // namespace tvm
-#endif // TVM_TARGET_SPIRV_IR_BUILDER_H_
+#endif // TVM_TARGET_VULKAN_IR_BUILDER_H_
diff --git a/src/target/spirv/spirv_support.cc b/src/target/vulkan/spirv_support.cc
similarity index 100%
rename from src/target/spirv/spirv_support.cc
rename to src/target/vulkan/spirv_support.cc
diff --git a/src/target/spirv/spirv_support.h b/src/target/vulkan/spirv_support.h
similarity index 98%
rename from src/target/spirv/spirv_support.h
rename to src/target/vulkan/spirv_support.h
index 83f9259511..f914f4d0d8 100644
--- a/src/target/spirv/spirv_support.h
+++ b/src/target/vulkan/spirv_support.h
@@ -23,8 +23,8 @@
* \brief Utility for determining which spirv capabilities a TVM
* target supports.
*/
-#ifndef TVM_TARGET_SPIRV_SPIRV_SUPPORT_H_
-#define TVM_TARGET_SPIRV_SPIRV_SUPPORT_H_
+#ifndef TVM_TARGET_VULKAN_SPIRV_SUPPORT_H_
+#define TVM_TARGET_VULKAN_SPIRV_SUPPORT_H_
#include <tvm/target/target.h>
#include <vulkan/vulkan_core.h>
@@ -294,4 +294,4 @@ struct SPIRVSupport {
} // namespace codegen
} // namespace tvm
-#endif // TVM_TARGET_SPIRV_SPIRV_SUPPORT_H_
+#endif // TVM_TARGET_VULKAN_SPIRV_SUPPORT_H_
diff --git a/src/target/spirv/spirv_utils.cc b/src/target/vulkan/spirv_utils.cc
similarity index 99%
rename from src/target/spirv/spirv_utils.cc
rename to src/target/vulkan/spirv_utils.cc
index df420a8f10..8a312d24dc 100644
--- a/src/target/spirv/spirv_utils.cc
+++ b/src/target/vulkan/spirv_utils.cc
@@ -37,7 +37,7 @@
#include <sstream>
#include <vector>
-#include "../../runtime/spirv/spirv_shader.h"
+#include "../../runtime/vulkan/spirv_shader.h"
#include "../../support/utils.h"
namespace tvm {
diff --git a/src/target/spirv/spirv_utils.h b/src/target/vulkan/spirv_utils.h
similarity index 88%
rename from src/target/spirv/spirv_utils.h
rename to src/target/vulkan/spirv_utils.h
index b441a559f8..03b98ea1b1 100644
--- a/src/target/spirv/spirv_utils.h
+++ b/src/target/vulkan/spirv_utils.h
@@ -16,8 +16,8 @@
* specific language governing permissions and limitations
* under the License.
*/
-#ifndef TVM_TARGET_SPIRV_SPIRV_UTILS_H_
-#define TVM_TARGET_SPIRV_SPIRV_UTILS_H_
+#ifndef TVM_TARGET_VULKAN_SPIRV_UTILS_H_
+#define TVM_TARGET_VULKAN_SPIRV_UTILS_H_
#include <tvm/ir/module.h>
#include <tvm/target/target.h>
@@ -26,7 +26,7 @@
#include <unordered_map>
#include <utility>
-#include "../../runtime/spirv/spirv_shader.h"
+#include "../../runtime/vulkan/spirv_shader.h"
namespace tvm {
namespace codegen {
@@ -43,4 +43,4 @@ std::pair<std::unordered_map<std::string, runtime::SPIRVShader>, std::string> Lo
} // namespace codegen
} // namespace tvm
-#endif // TVM_TARGET_SPIRV_SPIRV_UTILS_H_
+#endif // TVM_TARGET_VULKAN_SPIRV_UTILS_H_
diff --git a/src/target/vulkan/vulkan_fallback_module.cc b/src/target/vulkan/vulkan_fallback_module.cc
new file mode 100644
index 0000000000..0977085894
--- /dev/null
+++ b/src/target/vulkan/vulkan_fallback_module.cc
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file vulkan_fallback_module.cc
+ * \brief VulkanFallbackModuleNode — codegen-time placeholder used when the
+ * Vulkan runtime is not linked. Mirrors `VulkanModuleNode`'s
+ * save/load format byte-for-byte; see one-way comment in
+ * `SaveToBytes` below. Always compiled (independent of
+ * USE_VULKAN); never registered as an FFI factory or loader.
+ * Plain C++ — no Vulkan API types, no SPIR-V tooling deps.
+ */
+#include "vulkan_fallback_module.h"
+
+#include <tvm/ffi/extra/module.h>
+#include <tvm/ffi/function.h>
+
+#include <string>
+#include <utility>
+
+#include "../../support/bytes_io.h"
+
+namespace tvm {
+namespace target {
+
+class VulkanFallbackModuleNode : public ffi::ModuleObj {
+ public:
+ VulkanFallbackModuleNode(ffi::Map<ffi::String, ffi::Bytes> smap, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source)
+ : smap_(std::move(smap)),
+ fmt_(std::move(fmt)),
+ fmap_(std::move(fmap)),
+ source_(std::move(source)) {}
+
+ // Mirror the real module's kind so consumers cannot distinguish at the
+ // kind/api layer. Saved bytes load back as a real VulkanModuleNode on
+ // a Vulkan-equipped (USE_VULKAN=ON) receiver.
+ const char* kind() const final { return "vulkan"; }
+
+ int GetPropertyMask() const final { return ffi::Module::kBinarySerializable; }
+
+ ffi::Optional<ffi::Function> GetFunction(const ffi::String& name) final {
+ TVM_FFI_THROW(RuntimeError)
+ << "Vulkan runtime is not linked into this build; cannot launch kernels. "
+ << "Re-link with USE_VULKAN=ON or load this module in a Vulkan-equipped "
+ << "environment via tvm.runtime.load_module.";
+ TVM_FFI_UNREACHABLE();
+ }
+
+ ffi::Bytes SaveToBytes() const final {
+ // NOTE: serialization format MUST remain byte-identical to
+ // VulkanModuleNode::SaveToBytes in src/runtime/vulkan/vulkan_module.cc
+ // (the source of truth). Both produce a kind="vulkan" artifact that
+ // the loader (ffi.Module.load_from_bytes.vulkan, registered only when
+ // USE_VULKAN=ON) deserializes. If the real impl's format changes,
+ // mirror the change here. The dependency is one-way: this file
+ // follows; vulkan_module.cc does not reference this file.
+ //
+ // 3 fields only — the source map is in-memory inspection material and
+ // is NEVER serialized (matches upstream behavior for all backends).
+ // Each value in `smap_` is a self-packed SPIRVShader (flag + data
+ // vector); see src/runtime/vulkan/spirv_shader.h.
+ std::string buffer;
+ support::BytesOutStream stream(&buffer);
+ stream.Write(fmt_);
+ stream.Write(fmap_);
+ stream.Write(smap_);
+ return ffi::Bytes(std::move(buffer));
+ }
+
+ ffi::String InspectSource(const ffi::String& format) const final {
+ if (auto it = source_.find(format); it != source_.end()) {
+ return (*it).second;
+ }
+ if (format.empty()) {
+ // Default: aggregated SPIR-V text dump (key "spv").
+ if (auto it = source_.find("spv"); it != source_.end()) {
+ return (*it).second;
+ }
+ }
+ return ffi::String();
+ }
+
+ private:
+ // Per-kernel payload: kernel-name -> bytes. Each value is a
+ // serialized SPIRVShader (flag + uint32_t data segment); the runtime
+ // (USE_VULKAN=ON) deserializes via the inverse helper in
+ // src/runtime/vulkan/vulkan_module.cc. Multi-shader uniform
+ // Map<String, Bytes> across all multi-shader backends.
+ ffi::Map<ffi::String, ffi::Bytes> smap_;
+ // Format identifier — always "vulkan" today.
+ ffi::String fmt_;
+ // function information table.
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap_;
+ // In-memory source map for InspectSource — never serialized.
+ ffi::Map<ffi::String, ffi::String> source_;
+};
+
+ffi::Module VulkanFallbackModuleCreate(ffi::Map<ffi::String, ffi::Bytes> smap, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source) {
+ auto n = ffi::make_object<VulkanFallbackModuleNode>(std::move(smap), std::move(fmt),
+ std::move(fmap), std::move(source));
+ return ffi::Module(n);
+}
+
+} // namespace target
+} // namespace tvm
diff --git a/src/target/vulkan/vulkan_fallback_module.h b/src/target/vulkan/vulkan_fallback_module.h
new file mode 100644
index 0000000000..0113429db9
--- /dev/null
+++ b/src/target/vulkan/vulkan_fallback_module.h
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file vulkan_fallback_module.h
+ * \brief Codegen-facing Vulkan module factory.
+ *
+ * `VulkanModuleCreateWithFallback` is the ONLY entry point codegen uses
+ * to construct a Vulkan `ffi::Module`. It tries the runtime-registered
+ * factory "ffi.Module.create.vulkan" via the FFI registry; on miss it
+ * constructs a `VulkanFallbackModuleNode` directly. The fallback exists
+ * so that codegen can succeed on a build where the Vulkan runtime is
+ * not linked. This setup is helpful for cross compilation where we
+ * compile on one env and run on another.
+ */
+#ifndef TVM_TARGET_VULKAN_VULKAN_FALLBACK_MODULE_H_
+#define TVM_TARGET_VULKAN_VULKAN_FALLBACK_MODULE_H_
+
+#include <tvm/ffi/container/map.h>
+#include <tvm/ffi/extra/module.h>
+#include <tvm/ffi/function.h>
+
+#include <utility>
+
+#include "../../runtime/metadata.h"
+#include "../../support/env.h"
+
+namespace tvm {
+namespace target {
+
+/*!
+ * \brief Construct a `VulkanFallbackModuleNode` directly (no FFI registry
+ * round-trip). Used by `VulkanModuleCreateWithFallback` and by
+ * tests that explicitly want a fallback instance.
+ */
+ffi::Module VulkanFallbackModuleCreate(ffi::Map<ffi::String, ffi::Bytes> smap, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source);
+
+/*!
+ * \brief Codegen-time Vulkan module factory. Tries the FFI-registered
+ * "ffi.Module.create.vulkan"; on miss or when forced via env var,
+ * falls back to `VulkanFallbackModuleCreate`.
+ *
+ * - Registry hit (USE_VULKAN=ON build) → real `VulkanModuleNode` returned.
+ * - Registry miss (USE_VULKAN=OFF build) → `VulkanFallbackModuleNode`
+ * returned.
+ * - `TVM_COMPILE_FORCE_FALLBACK` env var truthy → fallback regardless of
+ * registry state (used by per-backend fallback tests on a USE_X=ON CI
+ * box).
+ */
+inline ffi::Module VulkanModuleCreateWithFallback(ffi::Map<ffi::String, ffi::Bytes> smap,
+ ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source) {
+ if (tvm::support::GetEnv<bool>("TVM_COMPILE_FORCE_FALLBACK", false)) {
+ return VulkanFallbackModuleCreate(std::move(smap), std::move(fmt), std::move(fmap),
+ std::move(source));
+ }
+ // Registry: "ffi.Module.create.vulkan" — real Vulkan runtime factory.
+ // Grep hint: grep -rn 'ffi.Module.create.vulkan' src/
+ auto fcreate = ffi::Function::GetGlobal("ffi.Module.create.vulkan");
+ if (fcreate.has_value()) {
+ return (*fcreate)(smap, fmt, fmap, source).cast<ffi::Module>();
+ }
+ return VulkanFallbackModuleCreate(std::move(smap), std::move(fmt), std::move(fmap),
+ std::move(source));
+}
+
+} // namespace target
+} // namespace tvm
+#endif // TVM_TARGET_VULKAN_VULKAN_FALLBACK_MODULE_H_
diff --git a/src/target/source/codegen_webgpu.cc b/src/target/webgpu/codegen_webgpu.cc
similarity index 93%
rename from src/target/source/codegen_webgpu.cc
rename to src/target/webgpu/codegen_webgpu.cc
index ca7b3878ae..2160d59a5b 100644
--- a/src/target/source/codegen_webgpu.cc
+++ b/src/target/webgpu/codegen_webgpu.cc
@@ -41,6 +41,7 @@
#include "../../runtime/thread_storage_scope.h"
#include "../../support/bytes_io.h"
#include "../build_common.h"
+#include "webgpu_fallback_module.h"
namespace tvm {
namespace codegen {
@@ -724,66 +725,21 @@ void CodeGenWebGPU::VisitStmt_(const WhileNode* op) {
stream << "}\n";
}
-//-------------------------------------------------
-// WebGPUSourceModule to enable export
-//-------------------------------------------------
-class WebGPUSourceModuleNode final : public ffi::ModuleObj {
- public:
- explicit WebGPUSourceModuleNode(std::unordered_map<std::string, std::string> smap,
- ffi::Map<ffi::String, runtime::FunctionInfo> fmap)
- : smap_(smap), fmap_(fmap) {}
-
- const char* kind() const final { return "webgpu"; }
- /*! \brief Get the property of the runtime module .*/
- int GetPropertyMask() const final { return ffi::Module::kBinarySerializable; }
-
- ffi::Optional<ffi::Function> GetFunction(const ffi::String& name) final {
- TVM_FFI_THROW(InternalError)
- << "WebGPUSourceModule is not directly runnable, export and run through tvmjs";
- return std::nullopt;
- }
-
- ffi::Bytes SaveToBytes() const final {
- std::string result;
- support::BytesOutStream stream(&result);
- stream.Write(fmap_);
- stream.Write(smap_);
- return ffi::Bytes(std::move(result));
- }
-
- ffi::String InspectSource(const ffi::String& format) const final {
- if (format == "func_info") {
- namespace json = ::tvm::ffi::json;
- json::Object obj;
- for (const auto& kv : fmap_) {
- obj.Set(kv.first, kv.second->SaveToJSON());
- }
- return std::string(json::Stringify(obj));
- } else {
- std::ostringstream os;
- for (auto kv : smap_) {
- os << kv.second;
- }
- return os.str();
- }
- }
-
- private:
- // function shader code table.
- std::unordered_map<std::string, std::string> smap_;
- // function information table.
- ffi::Map<ffi::String, runtime::FunctionInfo> fmap_;
-};
-
//-------------------------------------------------
// Build logic.
//-------------------------------------------------
+//
+// The "C++ side" canonical WebGPU module is `WebGPUFallbackModuleNode` in
+// src/target/webgpu/webgpu_fallback_module.{h,cc} — there is no native
+// WebGPU runtime in the C++ tree (the real receiver is the wasm runtime
+// in web/emcc/webgpu_runtime.cc).
ffi::Module BuildWebGPU(IRModule mod, Target target) {
mod = tirx::transform::PointerValueTypeRewrite()(std::move(mod));
bool output_ssa = false;
bool skip_readonly_decl = false;
- std::unordered_map<std::string, std::string> smap;
+ ffi::Map<ffi::String, ffi::Bytes> smap;
ffi::Map<ffi::String, runtime::FunctionInfo> fmap;
+ std::ostringstream source_maker;
// narrow all i64 to i32
mod = tirx::transform::ForceNarrowIndexToInt32()(std::move(mod));
@@ -803,11 +759,16 @@ ffi::Module BuildWebGPU(IRModule mod, Target target) {
cg.Init(output_ssa);
fmap.Set(f_name, cg.AddFunction(f, skip_readonly_decl));
std::string code = cg.Finish();
- smap[f_name] = code;
+ source_maker << "// Function: " << f_name << "\n" << code << "\n";
+ smap.Set(f_name, ffi::Bytes(std::move(code)));
}
- auto n = ffi::make_object<WebGPUSourceModuleNode>(smap, fmap);
- return ffi::Module(n);
+ // The aggregated WGSL source dump is preserved in the in-memory source
+ // map keyed by "wgsl" — only used by InspectSource and never serialized.
+ ffi::Map<ffi::String, ffi::String> source;
+ source.Set("wgsl", source_maker.str());
+ return target::WebGPUModuleCreateWithFallback(std::move(smap), ffi::String("wgsl"),
+ std::move(fmap), std::move(source));
}
TVM_FFI_STATIC_INIT_BLOCK() {
diff --git a/src/target/source/codegen_webgpu.h b/src/target/webgpu/codegen_webgpu.h
similarity index 95%
rename from src/target/source/codegen_webgpu.h
rename to src/target/webgpu/codegen_webgpu.h
index d0a541677a..750b51e5d2 100644
--- a/src/target/source/codegen_webgpu.h
+++ b/src/target/webgpu/codegen_webgpu.h
@@ -24,14 +24,14 @@
* This module generates WGSL shading language.
* See https://www.w3.org/TR/WGSL/ for the language reference.
*/
-#ifndef TVM_TARGET_SOURCE_CODEGEN_WEBGPU_H_
-#define TVM_TARGET_SOURCE_CODEGEN_WEBGPU_H_
+#ifndef TVM_TARGET_WEBGPU_CODEGEN_WEBGPU_H_
+#define TVM_TARGET_WEBGPU_CODEGEN_WEBGPU_H_
#include <tvm/target/codegen.h>
#include <string>
-#include "codegen_c.h"
+#include "../source/codegen_c.h"
namespace tvm {
namespace codegen {
@@ -104,4 +104,4 @@ class CodeGenWebGPU final : public CodeGenC {
} // namespace codegen
} // namespace tvm
-#endif // TVM_TARGET_SOURCE_CODEGEN_WEBGPU_H_
+#endif // TVM_TARGET_WEBGPU_CODEGEN_WEBGPU_H_
diff --git a/src/target/source/intrin_rule_webgpu.cc b/src/target/webgpu/intrin_rule_webgpu.cc
similarity index 100%
rename from src/target/source/intrin_rule_webgpu.cc
rename to src/target/webgpu/intrin_rule_webgpu.cc
diff --git a/src/target/webgpu/webgpu_fallback_module.cc b/src/target/webgpu/webgpu_fallback_module.cc
new file mode 100644
index 0000000000..574c210f77
--- /dev/null
+++ b/src/target/webgpu/webgpu_fallback_module.cc
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file webgpu_fallback_module.cc
+ * \brief WebGPUFallbackModuleNode — the canonical C++-side module for
+ * WebGPU. There is no native WebGPU runtime in the C++ tree; the
+ * real receiver is the wasm runtime in
+ * `web/emcc/webgpu_runtime.cc`, which registers only the
+ * load-from-bytes side of the FFI registry. The fallback's
+ * `SaveToBytes` produces the bytes that the wasm-side
+ * `WebGPUModuleLoadFromBytes` consumes.
+ *
+ * Always compiled (independent of any USE_X flag); never registers
+ * as an FFI factory or loader.
+ */
+#include "webgpu_fallback_module.h"
+
+#include <tvm/ffi/extra/json.h>
+#include <tvm/ffi/extra/module.h>
+#include <tvm/ffi/function.h>
+
+#include <sstream>
+#include <string>
+#include <utility>
+
+#include "../../support/bytes_io.h"
+
+namespace tvm {
+namespace target {
+
+class WebGPUFallbackModuleNode : public ffi::ModuleObj {
+ public:
+ WebGPUFallbackModuleNode(ffi::Map<ffi::String, ffi::Bytes> smap, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source)
+ : smap_(std::move(smap)),
+ fmt_(std::move(fmt)),
+ fmap_(std::move(fmap)),
+ source_(std::move(source)) {}
+
+ // The canonical C++-side WebGPU module — kind matches what the wasm
+ // runtime registers (`web/emcc/webgpu_runtime.cc`). Saved bytes load
+ // back as a real `WebGPUModuleNode` on the wasm side.
+ const char* kind() const final { return "webgpu"; }
+
+ int GetPropertyMask() const final { return ffi::Module::kBinarySerializable; }
+
+ ffi::Optional<ffi::Function> GetFunction(const ffi::String& name) final {
+ TVM_FFI_THROW(InternalError)
+ << "WebGPUFallbackModule is not directly runnable on the C++ side; "
+ << "export and run through tvmjs / the wasm WebGPU runtime.";
+ return std::nullopt;
+ }
+
+ ffi::Bytes SaveToBytes() const final {
+ // NOTE: serialization format MUST remain byte-identical to the
+ // wasm-side reader `WebGPUModuleLoadFromBytes` in
+ // web/emcc/webgpu_runtime.cc — which IS the receiver for WebGPU
+ // (there is no native C++ WebGPU runtime). The wasm reader expects
+ // 2 fields [fmap][smap] (no fmt — WebGPU is single-format "wgsl"
+ // today). If this format changes, the wasm-side reader MUST be
+ // updated in the same commit.
+ //
+ // Source map is in-memory inspection only and is NEVER serialized
+ // (matches cross-backend rule).
+ std::string result;
+ support::BytesOutStream stream(&result);
+ stream.Write(fmap_);
+ stream.Write(smap_);
+ return ffi::Bytes(std::move(result));
+ }
+
+ ffi::String InspectSource(const ffi::String& format) const final {
+ if (format == "func_info") {
+ namespace json = ::tvm::ffi::json;
+ json::Object obj;
+ for (const auto& kv : fmap_) {
+ obj.Set(kv.first, kv.second->SaveToJSON());
+ }
+ return std::string(json::Stringify(obj));
+ }
+ if (auto it = source_.find(format); it != source_.end()) {
+ return (*it).second;
+ }
+ if (format.empty()) {
+ if (auto it = source_.find("wgsl"); it != source_.end()) {
+ return (*it).second;
+ }
+ // Fall back to concatenated WGSL kernel sources (legacy upstream
+ // InspectSource behavior).
+ std::ostringstream os;
+ for (auto kv : smap_) {
+ os.write(kv.second.data(), kv.second.size());
+ }
+ return os.str();
+ }
+ return ffi::String();
+ }
+
+ private:
+ // Per-kernel payload: kernel-name -> WGSL source bytes. Multi-shader
+ // uniform Map<String, Bytes> (see plan "Per-kernel smap shape").
+ ffi::Map<ffi::String, ffi::Bytes> smap_;
+ // Format identifier ("wgsl" today — WebGPU is single-format). Not
+ // serialized in the on-disk bytes; the wasm receiver assumes "wgsl".
+ ffi::String fmt_;
+ // function information table.
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap_;
+ // In-memory source map for InspectSource — never serialized.
+ ffi::Map<ffi::String, ffi::String> source_;
+};
+
+ffi::Module WebGPUFallbackModuleCreate(ffi::Map<ffi::String, ffi::Bytes> smap, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source) {
+ auto n = ffi::make_object<WebGPUFallbackModuleNode>(std::move(smap), std::move(fmt),
+ std::move(fmap), std::move(source));
+ return ffi::Module(n);
+}
+
+} // namespace target
+} // namespace tvm
diff --git a/src/target/webgpu/webgpu_fallback_module.h b/src/target/webgpu/webgpu_fallback_module.h
new file mode 100644
index 0000000000..58e846c9ec
--- /dev/null
+++ b/src/target/webgpu/webgpu_fallback_module.h
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file webgpu_fallback_module.h
+ * \brief Codegen-facing WebGPU module factory.
+ *
+ * `WebGPUModuleCreateWithFallback` is the ONLY entry point codegen uses
+ * to construct a WebGPU `ffi::Module`. It tries the runtime-registered
+ * factory "ffi.Module.create.webgpu" via the FFI registry; on miss it
+ * constructs a `WebGPUFallbackModuleNode` directly. The fallback exists
+ * so that codegen can succeed on a build where the WebGPU runtime is
+ * not linked. This setup is helpful for cross compilation where we
+ * compile on one env and run on another.
+ */
+#ifndef TVM_TARGET_WEBGPU_WEBGPU_FALLBACK_MODULE_H_
+#define TVM_TARGET_WEBGPU_WEBGPU_FALLBACK_MODULE_H_
+
+#include <tvm/ffi/container/map.h>
+#include <tvm/ffi/extra/module.h>
+#include <tvm/ffi/function.h>
+
+#include <utility>
+
+#include "../../runtime/metadata.h"
+#include "../../support/env.h"
+
+namespace tvm {
+namespace target {
+
+/*!
+ * \brief Construct a `WebGPUFallbackModuleNode` directly (no FFI registry
+ * round-trip). Used by `WebGPUModuleCreateWithFallback` and by
+ * tests that explicitly want a fallback instance.
+ */
+ffi::Module WebGPUFallbackModuleCreate(ffi::Map<ffi::String, ffi::Bytes> smap, ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source);
+
+/*!
+ * \brief Codegen-time WebGPU module factory. Tries the FFI-registered
+ * "ffi.Module.create.webgpu"; on miss (always on the C++ side
+ * today — no native runtime) or when forced via env var, falls
+ * back to `WebGPUFallbackModuleCreate`.
+ *
+ * - Registry hit → real WebGPU module (currently never registered C++-
+ * side; wasm runtime registers only the load-from-bytes loader).
+ * - Registry miss → `WebGPUFallbackModuleNode` returned (the canonical
+ * C++ side for WebGPU).
+ * - `TVM_COMPILE_FORCE_FALLBACK` env var truthy → fallback regardless
+ * (test hook, no behavioral change here since fallback is always
+ * selected anyway).
+ */
+inline ffi::Module WebGPUModuleCreateWithFallback(ffi::Map<ffi::String, ffi::Bytes> smap,
+ ffi::String fmt,
+ ffi::Map<ffi::String, runtime::FunctionInfo> fmap,
+ ffi::Map<ffi::String, ffi::String> source) {
+ if (tvm::support::GetEnv<bool>("TVM_COMPILE_FORCE_FALLBACK", false)) {
+ return WebGPUFallbackModuleCreate(std::move(smap), std::move(fmt), std::move(fmap),
+ std::move(source));
+ }
+ // Registry: "ffi.Module.create.webgpu" — real WebGPU runtime factory.
+ // Grep hint: grep -rn 'ffi.Module.create.webgpu' src/
+ auto fcreate = ffi::Function::GetGlobal("ffi.Module.create.webgpu");
+ if (fcreate.has_value()) {
+ return (*fcreate)(smap, fmt, fmap, source).cast<ffi::Module>();
+ }
+ return WebGPUFallbackModuleCreate(std::move(smap), std::move(fmt), std::move(fmap),
+ std::move(source));
+}
+
+} // namespace target
+} // namespace tvm
+#endif // TVM_TARGET_WEBGPU_WEBGPU_FALLBACK_MODULE_H_
diff --git a/tests/python/codegen/test_target_codegen_cuda.py b/tests/python/codegen/test_target_codegen_cuda.py
index 31fa12d2cd..2567997098 100644
--- a/tests/python/codegen/test_target_codegen_cuda.py
+++ b/tests/python/codegen/test_target_codegen_cuda.py
@@ -39,7 +39,7 @@ def setup_cuda_compile_mode(request):
orig_func = tvm.contrib.nvcc.tvm_callback_cuda_compile
- def compile_mode_wrapper(code, target):
+ def compile_mode_wrapper(code):
if mode == "nvcc":
return tvm.contrib.nvcc.compile_cuda(code, target_format="fatbin", compiler="nvcc")
elif mode == "nvrtc":
@@ -1066,5 +1066,41 @@ def test_cuda_loop_step():
tvm.testing.assert_allclose(c_nd.numpy(), a_np + b_np)
+@tvm.testing.requires_gpu
+@tvm.testing.requires_cuda
+def test_export_load_with_fallback(monkeypatch, tmp_path):
+ """Force the codegen wrapper into the fallback branch, then export+load+run."""
+ n = 1024
+
+ @I.ir_module
+ class Module:
+ @T.prim_func
+ def main(A: T.Buffer((n,), "float32"), B: T.Buffer((n,), "float32")):
+ T.func_attr({"tirx.noalias": True})
+ for i_0 in T.thread_binding(n // 32, thread="blockIdx.x"):
+ for i_1 in T.thread_binding(32, thread="threadIdx.x"):
+ with T.sblock("B"):
+ v_i = T.axis.spatial(n, i_0 * 32 + i_1)
+ T.reads(A[v_i])
+ T.writes(B[v_i])
+ B[v_i] = A[v_i] + 1.0
+
+ monkeypatch.setenv("TVM_COMPILE_FORCE_FALLBACK", "1")
+ host_lib = tvm.compile(Module, target="cuda")
+ monkeypatch.delenv("TVM_COMPILE_FORCE_FALLBACK")
+
+ lib_path = str(tmp_path / "lib.so")
+ host_lib.export_library(lib_path)
+ reloaded = tvm.runtime.load_module(lib_path)
+
+ dev = tvm.cuda(0)
+ a_np = np.random.uniform(size=(n,)).astype("float32")
+ b_np = np.zeros((n,), dtype="float32")
+ a = tvm.runtime.tensor(a_np, dev)
+ b = tvm.runtime.tensor(b_np, dev)
+ reloaded["main"](a, b)
+ np.testing.assert_allclose(b.numpy(), a_np + 1.0, rtol=1e-5)
+
+
if __name__ == "__main__":
tvm.testing.main()
diff --git a/tests/python/codegen/test_target_codegen_metal.py b/tests/python/codegen/test_target_codegen_metal.py
index 781fa15d01..4f8ab4efdd 100644
--- a/tests/python/codegen/test_target_codegen_metal.py
+++ b/tests/python/codegen/test_target_codegen_metal.py
@@ -208,5 +208,31 @@ def test_func_with_trailing_pod_params():
assert occurrences == 1, occurrences
+@tvm.testing.requires_metal(support_required="compile-only")
+def test_export_load_with_fallback(monkeypatch, tmp_path):
+ """Force the codegen wrapper into the fallback branch, then export."""
+ n = 1024
+
+ @I.ir_module
+ class Module:
+ @T.prim_func
+ def main(A: T.Buffer((n,), "float32"), B: T.Buffer((n,), "float32")):
+ T.func_attr({"tirx.noalias": True})
+ for i_0 in T.thread_binding(n // 32, thread="blockIdx.x"):
+ for i_1 in T.thread_binding(32, thread="threadIdx.x"):
+ with T.sblock("B"):
+ v_i = T.axis.spatial(n, i_0 * 32 + i_1)
+ T.reads(A[v_i])
+ T.writes(B[v_i])
+ B[v_i] = A[v_i] + 1.0
+
+ monkeypatch.setenv("TVM_COMPILE_FORCE_FALLBACK", "1")
+ host_lib = tvm.compile(Module, target="metal")
+ monkeypatch.delenv("TVM_COMPILE_FORCE_FALLBACK")
+
+ lib_path = str(tmp_path / "lib.so")
+ host_lib.export_library(lib_path)
+
+
if __name__ == "__main__":
tvm.testing.main()
diff --git a/tests/python/codegen/test_target_codegen_opencl.py b/tests/python/codegen/test_target_codegen_opencl.py
index e3c76d475d..b6367006eb 100644
--- a/tests/python/codegen/test_target_codegen_opencl.py
+++ b/tests/python/codegen/test_target_codegen_opencl.py
@@ -265,5 +265,43 @@ def _get_maximum_kernel_args(source):
return max_args
+@tvm.testing.requires_gpu
+@tvm.testing.requires_opencl
+def test_export_load_with_fallback(monkeypatch, tmp_path):
+ """Force the codegen wrapper into the fallback branch, then export+load+run."""
+ import numpy as np
+
+ n = 1024
+
+ @I.ir_module
+ class Module:
+ @T.prim_func
+ def main(A: T.Buffer((n,), "float32"), B: T.Buffer((n,), "float32")):
+ T.func_attr({"tirx.noalias": True})
+ for i_0 in T.thread_binding(n // 32, thread="blockIdx.x"):
+ for i_1 in T.thread_binding(32, thread="threadIdx.x"):
+ with T.sblock("B"):
+ v_i = T.axis.spatial(n, i_0 * 32 + i_1)
+ T.reads(A[v_i])
+ T.writes(B[v_i])
+ B[v_i] = A[v_i] + 1.0
+
+ monkeypatch.setenv("TVM_COMPILE_FORCE_FALLBACK", "1")
+ host_lib = tvm.compile(Module, target=target)
+ monkeypatch.delenv("TVM_COMPILE_FORCE_FALLBACK")
+
+ lib_path = str(tmp_path / "lib.so")
+ host_lib.export_library(lib_path)
+ reloaded = tvm.runtime.load_module(lib_path)
+
+ dev = tvm.device(target, 0)
+ a_np = np.random.uniform(size=(n,)).astype("float32")
+ b_np = np.zeros((n,), dtype="float32")
+ a = tvm.runtime.tensor(a_np, dev)
+ b = tvm.runtime.tensor(b_np, dev)
+ reloaded["main"](a, b)
+ np.testing.assert_allclose(b.numpy(), a_np + 1.0, rtol=1e-5)
+
+
if __name__ == "__main__":
tvm.testing.main()
diff --git a/tests/python/codegen/test_target_codegen_rocm.py b/tests/python/codegen/test_target_codegen_rocm.py
index 2b62f0c08b..9e4f9f1bc1 100644
--- a/tests/python/codegen/test_target_codegen_rocm.py
+++ b/tests/python/codegen/test_target_codegen_rocm.py
@@ -152,3 +152,42 @@ def test_rocm_vectorized_exp():
b = tvm.runtime.tensor(np.zeros((4,)).astype("float32"), dev)
mod(a, b)
tvm.testing.assert_allclose(b.numpy(), np.exp2(a.numpy()))
+
+
+@tvm.testing.requires_rocm
+def test_export_load_with_fallback(monkeypatch, tmp_path):
+ """Force the codegen wrapper into the fallback branch, then export+load+run."""
+ n = 1024
+
+ @I.ir_module
+ class Module:
+ @T.prim_func
+ def main(A: T.Buffer((n,), "float32"), B: T.Buffer((n,), "float32")):
+ T.func_attr({"tirx.noalias": True})
+ for i_0 in T.thread_binding(n // 32, thread="blockIdx.x"):
+ for i_1 in T.thread_binding(32, thread="threadIdx.x"):
+ with T.sblock("B"):
+ v_i = T.axis.spatial(n, i_0 * 32 + i_1)
+ T.reads(A[v_i])
+ T.writes(B[v_i])
+ B[v_i] = A[v_i] + 1.0
+
+ monkeypatch.setenv("TVM_COMPILE_FORCE_FALLBACK", "1")
+ host_lib = tvm.compile(Module, target="rocm")
+ monkeypatch.delenv("TVM_COMPILE_FORCE_FALLBACK")
+
+ lib_path = str(tmp_path / "lib.so")
+ host_lib.export_library(lib_path)
+ reloaded = tvm.runtime.load_module(lib_path)
+
+ dev = tvm.rocm(0)
+ a_np = np.random.uniform(size=(n,)).astype("float32")
+ b_np = np.zeros((n,), dtype="float32")
+ a = tvm.runtime.tensor(a_np, dev)
+ b = tvm.runtime.tensor(b_np, dev)
+ reloaded["main"](a, b)
+ np.testing.assert_allclose(b.numpy(), a_np + 1.0, rtol=1e-5)
+
+
+if __name__ == "__main__":
+ tvm.testing.main()
diff --git a/tests/python/codegen/test_target_codegen_vulkan.py b/tests/python/codegen/test_target_codegen_vulkan.py
index e6a24061c8..b48f6f203e 100644
--- a/tests/python/codegen/test_target_codegen_vulkan.py
+++ b/tests/python/codegen/test_target_codegen_vulkan.py
@@ -592,5 +592,31 @@ def test_unary():
run_test(*func)
+@tvm.testing.requires_vulkan(support_required="compile-only")
+def test_export_load_with_fallback(monkeypatch, tmp_path):
+ """Force the codegen wrapper into the fallback branch, then export."""
+ n = 1024
+
+ @I.ir_module
+ class Module:
+ @T.prim_func
+ def main(A: T.Buffer((n,), "float32"), B: T.Buffer((n,), "float32")):
+ T.func_attr({"tirx.noalias": True})
+ for i_0 in T.thread_binding(n // 32, thread="blockIdx.x"):
+ for i_1 in T.thread_binding(32, thread="threadIdx.x"):
+ with T.sblock("B"):
+ v_i = T.axis.spatial(n, i_0 * 32 + i_1)
+ T.reads(A[v_i])
+ T.writes(B[v_i])
+ B[v_i] = A[v_i] + 1.0
+
+ monkeypatch.setenv("TVM_COMPILE_FORCE_FALLBACK", "1")
+ host_lib = tvm.compile(Module, target="vulkan")
+ monkeypatch.delenv("TVM_COMPILE_FORCE_FALLBACK")
+
+ lib_path = str(tmp_path / "lib.so")
+ host_lib.export_library(lib_path)
+
+
if __name__ == "__main__":
tvm.testing.main()