You are viewing a plain text version of this content. The canonical link for it is here.
Posted to discuss-archive@tvm.apache.org by "tqchen (via GitHub)" <gi...@apache.org> on 2026/07/05 01:57:50 UTC
[PR] [TEST] Run the Python test suite in parallel with a shared GPU lock [tvm-ffi]
tqchen opened a new pull request, #654:
URL: https://github.com/apache/tvm-ffi/pull/654
## Rationale
The Python test suite runs serially today. Running it in parallel with
`pytest-xdist` (`-n auto`) across multiple worker processes cuts wall-clock
time substantially. The one obstacle is the single machine-local GPU: tests
that execute device work cannot run concurrently on it safely.
## Changes
- Add a `tvm_ffi.testing.run_with_gpu_lock` helper that serializes device work
through an advisory file lock shared across all worker processes (built on
`tvm_ffi.utils.FileLock`). The lock lives under the system temp dir by
default and can be redirected with the `TVM_FFI_TEST_LOCK_DIR` environment
variable.
- Wrap the device-executing regions of the GPU-touching tests with the helper
so they serialize on the shared lock, while source compilation stays outside
the lock and continues to run in parallel.
- Add `pytest-xdist` to the `test` dependency group and enable `-n auto` via
`addopts`. Pass `-n0` to opt out of parallelism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org
Re: [PR] [TEST] Run the Python test suite in parallel with a shared GPU lock [tvm-ffi]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #654:
URL: https://github.com/apache/tvm-ffi/pull/654#discussion_r3524157996
##########
python/tvm_ffi/testing/_locking.py:
##########
@@ -0,0 +1,100 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Helpers for tests that use exclusive machine-local resources.
+
+Parallel test runners (for example ``pytest -n auto``) spread tests across
+several worker processes. Tests that reach for a shared, machine-local
+resource such as a single GPU must serialize with each other so that
+concurrent access does not exhaust device memory or corrupt device state.
+:func:`run_with_gpu_lock` provides that serialization through an advisory
+file lock shared by every cooperating worker on the machine.
+"""
+
+from __future__ import annotations
+
+import os
+import tempfile
+from collections.abc import Callable
+from pathlib import Path
+from typing import Any, TypeVar
+
+from ..utils import FileLock
+
+_LOCK_DIR_ENV_VAR = "TVM_FFI_TEST_LOCK_DIR"
+_LOCK_DIR_NAME = "tvm-ffi-testing-locks"
Review Comment:

On shared development servers or multi-user CI/CD environments, using a static, predictable directory name like `tvm-ffi-testing-locks` under the system temporary directory (`/tmp` on Unix) can lead to `PermissionError` or flaky test runs. The first user who runs the tests will create the directory with their own ownership and restrictive permissions (due to default umask). When a second user subsequently runs the tests, they will be unable to write to or create the lock file inside that directory, causing the test suite to fail.
To prevent this, we should make the default lock directory user-specific by appending the current username (retrieved safely via `getpass.getuser()`).
```python
import getpass
import os
import tempfile
from collections.abc import Callable
from pathlib import Path
from typing import Any, TypeVar
from ..utils import FileLock
_LOCK_DIR_ENV_VAR = "TVM_FFI_TEST_LOCK_DIR"
try:
_username = getpass.getuser()
except Exception:
_username = "unknown"
_LOCK_DIR_NAME = f"tvm-ffi-testing-locks-{_username}"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org
Re: [PR] [TEST] Run the Python test suite in parallel with a shared GPU lock [tvm-ffi]
Posted by "tqchen (via GitHub)" <gi...@apache.org>.
tqchen commented on PR #654:
URL: https://github.com/apache/tvm-ffi/pull/654#issuecomment-4886156451
Benchmarked the Python test suite locally at exact PR head `e308936185d8accc4ba0cf4465aba2c807bd26ae`.
Environment: AMD EPYC 7413 (24 physical / 48 logical CPUs), one NVIDIA H200, driver 595.58.03, Python 3.12.12, pytest 9.1.1, pytest-xdist 3.8.0, and torch 2.12.1+cu130. The configured `-n auto` resolved to **48 workers** (`created: 48/48 workers`).
Compared the same checkout and environment:
- Serial: `pytest -n 0 tests/python`
- Configured parallel: `pytest tests/python` (uses the PR's `addopts = ["-n", "auto"]`)
Every reported run completed with the same result: **2333 passed, 7 skipped, 2 xfailed** (2342 items). The seven skips are environment-dependent and identical in both modes: three require CUDA > 13.0, two require Python 3.13+, one requires free-threaded Python, and one requires ROCm.
| Cache condition | Repetitions per mode | Serial wall samples | `-n auto` wall samples | Median result |
|---|---:|---|---|---|
| Native-extension cache warm | 5 | 27.758, 27.591, 26.975, 26.459, 28.811 s | 24.391, 25.917, 24.292, 25.806, 24.497 s | 27.591 -> 24.497 s: **1.126x**, **11.2% lower wall time** |
| Fresh native-extension cache per invocation | 3 | 97.253, 96.633, 96.993 s | 39.883, 43.168, 39.811 s | 96.993 -> 39.883 s: **2.432x**, **58.9% lower wall time** |
Method and caveats:
- Wall time is external shell time, so it includes collection and xdist worker startup.
- The warm-cache comparison discarded one warm-up per mode, then used balanced alternating order (AB/BA) for five measurements each.
- Each cold-cache invocation received fresh, disjoint `TVM_FFI_CACHE_DIR` and `TORCH_EXTENSIONS_DIR` paths, so neither mode warmed the other. Installed dependencies and OS page caches remained warm; "cold" here specifically means the content-addressed native-extension cache, not a completely cold machine.
- The 48-worker run trades resources for latency. Median aggregate user CPU time increased from 64.654 s to 490.378 s (7.58x) in the warm-cache comparison and from 109.493 s to 538.752 s (4.92x) in the cold-cache comparison.
- These are single-node results. The exact gain will vary with worker count, CPU allocation, dependency versions, and cache state.
On this host, parallel execution has a modest steady-state benefit and a large clean extension-build benefit, while retaining identical test outcomes across all measured runs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org
Re: [PR] [TEST] Run the Python test suite in parallel with a shared GPU lock [tvm-ffi]
Posted by "tqchen (via GitHub)" <gi...@apache.org>.
tqchen merged PR #654:
URL: https://github.com/apache/tvm-ffi/pull/654
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org