You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/03/26 06:19:40 UTC
(spark) branch master updated: [SPARK-56067][PYTHON] Lazy import psutil to improve import speed
This is an automated email from the ASF dual-hosted git repository.
ruifengz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 649499191043 [SPARK-56067][PYTHON] Lazy import psutil to improve import speed
649499191043 is described below
commit 64949919104315e9aa27b22180bf2e5b419a9514
Author: Tian Gao <ga...@hotmail.com>
AuthorDate: Thu Mar 26 14:19:26 2026 +0800
[SPARK-56067][PYTHON] Lazy import psutil to improve import speed
### What changes were proposed in this pull request?
Instead of `import psutil` at module-level, we do it in the function to avoid always triggering the import when we `import pyspark.`
### Why are the changes needed?
`psutil` itself is relatively fast to import, but it's also unnecessary. More importantly, it's the only outlier after `numpy` and memory_profiler`. After getting rid of it, we can write a test to check if we import 3rd party library when we do `import pyspark` to prevent similar issues in the future.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
`test_shuffle` passed locally, the rest is on CI.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #54897 from gaogaotiantian/lazy-import-psutil.
Authored-by: Tian Gao <ga...@hotmail.com>
Signed-off-by: Ruifeng Zheng <ru...@apache.org>
---
python/pyspark/shuffle.py | 21 ++++++++++++---------
1 file changed, 12 insertions(+), 9 deletions(-)
diff --git a/python/pyspark/shuffle.py b/python/pyspark/shuffle.py
index 30147afe0086..dc43353e36db 100644
--- a/python/pyspark/shuffle.py
+++ b/python/pyspark/shuffle.py
@@ -35,13 +35,19 @@ from pyspark.serializers import (
)
from pyspark.util import fail_on_stopiteration
-try:
- import psutil
+process = None
- process = None
- def get_used_memory():
- """Return the used memory in MiB"""
+def get_used_memory():
+ """Return the used memory in MiB"""
+ try:
+ import psutil
+
+ has_psutil = True
+ except ImportError:
+ has_psutil = False
+
+ if has_psutil:
global process
if process is None or process._pid != os.getpid():
process = psutil.Process(os.getpid())
@@ -51,10 +57,7 @@ try:
info = process.get_memory_info()
return info.rss >> 20
-except ImportError:
-
- def get_used_memory():
- """Return the used memory in MiB"""
+ else:
if platform.system() == "Linux":
for line in open("/proc/self/status"):
if line.startswith("VmRSS:"):
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org