You are viewing a plain text version of this content. The canonical link for it is here.
Posted to discuss-archive@tvm.apache.org by "tlopex (via GitHub)" <gi...@apache.org> on 2026/06/11 20:52:09 UTC

[PR] [Runtime][Disco] Fix Disco session attribute storage and test gating [tvm]

tlopex opened a new pull request, #19736:
URL: https://github.com/apache/tvm/pull/19736

   Problem
   -------
   1. Disco sessions could no longer be constructed. The tvm_ffi Object metaclass now gives every subclass `__slots__ = ()`, so the Python-side attribute assignments the Disco wrappers depend on raise AttributeError: DPackedFunc/DModule store their owning `session`, and Session caches method lookups in `self.__dict__`.
   
   2. The disco tests did not gate on the disco runtime actually being present. A build can report USE_NCCL=ON / USE_NVSHMEM=ON yet not ship the disco CCL/NVSHMEM runtime (the pip wheel is exactly this case). On such builds the disco tests errored during collection or skipped on the wrong signal instead of skipping cleanly. Conversely, several tests silently passed or collected nothing while exercising no code, so the gaps went unnoticed.
   
   3. TVM could not even be built with USE_NVSHMEM=ON: src/runtime/extra/contrib/nvshmem/dist_gemm.cu failed to compile (GetDataSize / LOG / FATAL undefined after the tvm-ffi migration).
   
   Fix
   ---
   Session attribute storage:
   - Opt DPackedFunc, DModule, and Session back into instance attributes via the documented `__slots__ = ("__dict__",)` opt-out.
   
   NVSHMEM runtime build:
   - dist_gemm.cu: `GetDataSize` now lives in namespace tvm::ffi, and the `if (worker == nullptr) LOG(FATAL)` checks referenced macros that were out of scope. Use ffi::GetDataSize and TVM_FFI_ICHECK(worker != nullptr). LOG(FATAL) here already throws ffi::Error("InternalError", ...), so this is equivalent and matches the file's existing tvm-ffi style.
   
   Test gating:
   - Add a `tvm.testing.requires_nvshmem` feature that probes the `runtime.disco.nvshmem.init_nvshmem_uid` runtime function rather than the cmake flag.
   - test_loader.py skips at module level unless `runtime.disco.compiled_ccl` is present and reports "nccl" (the same source of truth as test_ccl.py), and requires two GPUs via tvm.testing.requires_multi_gpu.
   - test_nvshmem.py gates every test on NVSHMEM support and on the machine having one CUDA device per worker. Each body now runs in a freshly spawned process with a timeout, because nvshmem_init may be called only once per process and disco worker-0 lives in the test process, yet pytest previously ran every body in that one process. Replaces the hand-rolled __main__ matrix with tvm.testing.main().
   - Move the socket-session helpers duplicated verbatim by test_session.py and test_nvshmem.py into tvm.testing.disco (following the tvm.testing.tir precedent). The module stays importable when `tvm.runtime.disco` is None, so dependent test modules skip instead of erroring at collection.
   
   Bugs surfaced and fixed while validating the above:
   - Socket-session deadlock: test_session.py hung indefinitely once a second socket-session test ran. create_socket_session relied on GC/__del__ to shut down the previous SocketSessionTester, but a failing test kept the old tester alive through its traceback, so two socket sessions ran at once and the new SocketSession server thread blocked forever waiting for the remote node. Add an idempotent SocketSessionTester.shutdown() and call it explicitly before constructing the next tester. The shared tester also launches remote workers with `sys.executable` instead of hard-coded "python3", surfaces SocketSession constructor failures from the server thread, and initializes the attributes used by teardown first so it is safe when __init__ raises.
   - test_callback.py hung at interpreter teardown: the test created a ProcessSession with NCCL but never shut it down, so garbage-collecting the session at exit deadlocked. Shut the session down explicitly in a finally block.
   - test_custom_allreduce.py silently skipped all 12 test_allreduce cases: `compiled_ccl()` returns the backend name string, but the code iterated over it as if it were a list ('n','c','c','l'), so the `ccl` parametrization was always empty. Compare the value directly, matching test_loader.py.
   - test_session.py::test_num_workers used a bare `return` to skip the socket-session/num_workers<2 case, which reported as PASSED and hid the lost coverage. Use pytest.skip with a reason.
   - test_nvshmem.py: factor the init/finalize boilerplate, duplicated in four test bodies, into _init_nvshmem()/_finalize_nvshmem() helpers.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [PR] [Runtime][Disco] Fix Disco session attribute storage and test gating [tvm]

Posted by "tqchen (via GitHub)" <gi...@apache.org>.
tqchen commented on code in PR #19736:
URL: https://github.com/apache/tvm/pull/19736#discussion_r3399526973


##########
python/tvm/runtime/disco/session.py:
##########
@@ -103,6 +109,10 @@ class Session(Object):
     """A Disco interactive session. It allows users to interact with the Disco command queue with
     various PackedFunc calling convention."""
 
+    # Opt out of the default `__slots__ = ()` of tvm_ffi Object subclasses, as
+    # the session caches method lookups in dynamic instance attributes.
+    __slots__ = ("__dict__",)

Review Comment:
   better approach is to add slot session and _cache



##########
python/tvm/runtime/disco/session.py:
##########
@@ -78,6 +78,10 @@ def debug_copy_from(
 class DPackedFunc(DRef):
     """A PackedFunc in a Disco session."""
 
+    # Opt out of the default `__slots__ = ()` of tvm_ffi Object subclasses to
+    # allow Python-side instance attributes (the associated `session`).
+    __slots__ = ("__dict__",)

Review Comment:
   better approach is to add slot session



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [PR] [Runtime][Disco] Fix Disco session attribute storage and test gating [tvm]

Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #19736:
URL: https://github.com/apache/tvm/pull/19736#discussion_r3399100952


##########
python/tvm/testing/disco.py:
##########
@@ -0,0 +1,136 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Testing utilities for the Disco distributed runtime."""
+
+# Defer annotation evaluation: `tvm.runtime.disco` is None on builds without
+# the disco runtime, and this module must still be importable there.
+from __future__ import annotations
+
+import socket
+import subprocess
+import sys
+import threading
+
+from tvm.runtime import disco as di
+
+_SOCKET_SESSION_TESTER = None
+
+
+def _get_free_port() -> int:
+    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+    sock.bind(("", 0))
+    port = sock.getsockname()[1]
+    sock.close()
+    return port
+
+
+class SocketSessionTester:
+    """Run a disco SocketSession with one local node and remote nodes.
+
+    Each remote node is a `tvm.exec.disco_remote_socket_session` subprocess
+    launched with the current Python interpreter.
+    """
+
+    def __init__(self, num_workers, num_nodes=2, num_groups=1):
+        # Initialize the attributes used by __del__ first, so that teardown is
+        # safe even when __init__ raises below.
+        self.sess = None
+        self.remote_nodes = []
+        assert num_workers % num_nodes == 0
+        num_workers_per_node = num_workers // num_nodes
+        server_host = "localhost"
+        server_port = _get_free_port()
+        server_exc = []
+
+        def start_server():
+            try:
+                self.sess = di.SocketSession(
+                    num_nodes, num_workers_per_node, num_groups, server_host, server_port
+                )
+            except Exception as exc:  # pylint: disable=broad-except
+                server_exc.append(exc)
+
+        thread = threading.Thread(target=start_server)
+        thread.start()
+
+        cmd = "tvm.exec.disco_remote_socket_session"
+        for _i in range(num_nodes - 1):
+            self.remote_nodes.append(
+                subprocess.Popen(
+                    [
+                        sys.executable,
+                        "-m",
+                        cmd,
+                        server_host,
+                        str(server_port),
+                        str(num_workers_per_node),
+                    ],
+                    stdout=sys.stdout,
+                    stderr=sys.stderr,
+                )
+            )
+
+        thread.join()
+        if server_exc:
+            raise server_exc[0]

Review Comment:
   ![high](https://www.gstatic.com/codereviewagent/high-priority.svg)
   
   If any of the remote node subprocesses fail to start or crash immediately (e.g., due to import errors or missing dependencies), the `SocketSession` server will block indefinitely waiting for connections. A bare `thread.join()` will then hang the entire test suite. Adding a timeout to `thread.join()` and cleaning up the launched subprocesses on timeout prevents indefinite hangs in CI.
   
   ```python
           thread.join(timeout=60)
           if thread.is_alive():
               for node in self.remote_nodes:
                   node.kill()
                   node.wait()
               raise RuntimeError(
                   f'SocketSession server thread timed out after 60 seconds waiting for '
                   f'{num_nodes - 1} remote node(s) to connect.'
               )
           if server_exc:
               raise server_exc[0]
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [PR] [Runtime][Disco] Fix Disco session attribute storage and test gating [tvm]

Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #19736:
URL: https://github.com/apache/tvm/pull/19736#discussion_r3399100939


##########
python/tvm/testing/disco.py:
##########
@@ -0,0 +1,136 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Testing utilities for the Disco distributed runtime."""
+
+# Defer annotation evaluation: `tvm.runtime.disco` is None on builds without
+# the disco runtime, and this module must still be importable there.
+from __future__ import annotations
+
+import socket
+import subprocess
+import sys
+import threading
+
+from tvm.runtime import disco as di
+
+_SOCKET_SESSION_TESTER = None
+
+
+def _get_free_port() -> int:
+    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+    sock.bind(("", 0))
+    port = sock.getsockname()[1]
+    sock.close()
+    return port
+
+
+class SocketSessionTester:
+    """Run a disco SocketSession with one local node and remote nodes.
+
+    Each remote node is a `tvm.exec.disco_remote_socket_session` subprocess
+    launched with the current Python interpreter.
+    """
+
+    def __init__(self, num_workers, num_nodes=2, num_groups=1):
+        # Initialize the attributes used by __del__ first, so that teardown is
+        # safe even when __init__ raises below.
+        self.sess = None
+        self.remote_nodes = []
+        assert num_workers % num_nodes == 0
+        num_workers_per_node = num_workers // num_nodes
+        server_host = "localhost"
+        server_port = _get_free_port()
+        server_exc = []
+
+        def start_server():
+            try:
+                self.sess = di.SocketSession(
+                    num_nodes, num_workers_per_node, num_groups, server_host, server_port
+                )
+            except Exception as exc:  # pylint: disable=broad-except
+                server_exc.append(exc)
+
+        thread = threading.Thread(target=start_server)
+        thread.start()

Review Comment:
   ![high](https://www.gstatic.com/codereviewagent/high-priority.svg)
   
   The server thread is not configured as a daemon thread. If the main thread or any subprocess fails or exits with an exception, this non-daemon thread can block indefinitely and prevent the test process from exiting, leading to CI hangs. Setting `daemon=True` ensures the thread does not block process exit.
   
   ```suggestion
           thread = threading.Thread(target=start_server, daemon=True)
           thread.start()
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [PR] [Runtime][Disco] Fix session attribute storage, NVSHMEM build, and test gating [tvm]

Posted by "tqchen (via GitHub)" <gi...@apache.org>.
tqchen merged PR #19736:
URL: https://github.com/apache/tvm/pull/19736


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org