You are viewing a plain text version of this content. The canonical link for it is here.
Posted to discuss-archive@tvm.apache.org by "tqchen (via GitHub)" <gi...@apache.org> on 2026/07/04 19:54:26 UTC
[PR] [TEST] Serialize local GPU execution under pytest-xdist [tvm]
tqchen opened a new pull request, #19942:
URL: https://github.com/apache/tvm/pull/19942
Add tvm.testing.run_with_gpu_lock backed by the existing tvm_ffi.utils.FileLock. Migrate live local GPU tests to acquire the machine-local lock around device execution, synchronization, host transfer, and checks while leaving target construction and compilation outside the critical section.
Replace the custom xdist scheduler with standard xdist_group placement for the order-dependent test family. RPC tests retain dynamic port allocation and per-test process isolation rather than gaining a broad category lock.
Validation: all changed Python files pass pre-commit and bytecode compilation. On one H200, the 70-file changed-test selection with --dist=loadgroup scaled from 1446 seconds at one worker to 196 seconds at 16 workers; 24 workers took 208 seconds. No lock timeout, deadlock, or worker loss occurred, and the known baseline outcomes reproduced across worker counts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org
Re: [PR] [TEST] Serialize local GPU execution under pytest-xdist [tvm]
Posted by "tqchen (via GitHub)" <gi...@apache.org>.
tqchen merged PR #19942:
URL: https://github.com/apache/tvm/pull/19942
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org
Re: [PR] [TEST] Serialize local GPU execution under pytest-xdist [tvm]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #19942:
URL: https://github.com/apache/tvm/pull/19942#discussion_r3523782055
##########
python/tvm/testing/locking.py:
##########
@@ -0,0 +1,65 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Helpers for tests that use exclusive machine resources."""
+
+import os
+import tempfile
+from collections.abc import Callable
+from pathlib import Path
+from typing import Any, TypeVar
+
+from tvm_ffi.utils import FileLock
+
+_LOCK_DIR_ENV_VAR = "TVM_TEST_LOCK_DIR"
+_LOCK_DIR_NAME = "tvm-testing-locks"
Review Comment:

On shared multi-user systems (such as shared development servers or CI runners), using a fixed directory name like `tvm-testing-locks` under the shared temporary directory (`/tmp`) can lead to `PermissionError` or access conflicts if the directory is created by one user and another user attempts to run tests.
To prevent these multi-user permission conflicts, it is highly recommended to append the current username to the lock directory name.
```suggestion
import getpass
_LOCK_DIR_NAME = f"tvm-testing-locks-{getpass.getuser()}"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org
Re: [PR] [TEST] Serialize local GPU execution under pytest-xdist [tvm]
Posted by "tqchen (via GitHub)" <gi...@apache.org>.
tqchen commented on code in PR #19942:
URL: https://github.com/apache/tvm/pull/19942#discussion_r3523785829
##########
python/tvm/testing/locking.py:
##########
@@ -0,0 +1,65 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Helpers for tests that use exclusive machine resources."""
+
+import os
+import tempfile
+from collections.abc import Callable
+from pathlib import Path
+from typing import Any, TypeVar
+
+from tvm_ffi.utils import FileLock
+
+_LOCK_DIR_ENV_VAR = "TVM_TEST_LOCK_DIR"
+_LOCK_DIR_NAME = "tvm-testing-locks"
Review Comment:
Thanks for flagging the permission failure mode. I did not add a username suffix because two users sharing one physical GPU would then lock different files and enter the critical section concurrently. The default path is intentionally a same-user coordination domain. A multi-user runner must point every contender at one administrator-provisioned TVM_TEST_LOCK_DIR with a shared-writable persistent lock file, or provide GPU exclusivity in the runner.
Commit 66422f7256 documents that contract explicitly and also aligns the example callback name with run_and_check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org