You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/03/26 06:19:40 UTC

(spark) branch master updated: [SPARK-56067][PYTHON] Lazy import psutil to improve import speed

This is an automated email from the ASF dual-hosted git repository.

ruifengz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 649499191043 [SPARK-56067][PYTHON] Lazy import psutil to improve import speed
649499191043 is described below

commit 64949919104315e9aa27b22180bf2e5b419a9514
Author: Tian Gao <ga...@hotmail.com>
AuthorDate: Thu Mar 26 14:19:26 2026 +0800

    [SPARK-56067][PYTHON] Lazy import psutil to improve import speed
    
    ### What changes were proposed in this pull request?
    
    Instead of `import psutil` at module-level, we do it in the function to avoid always triggering the import when we `import pyspark.`
    
    ### Why are the changes needed?
    
    `psutil` itself is relatively fast to import, but it's also unnecessary. More importantly, it's the only outlier after `numpy` and memory_profiler`. After getting rid of it, we can write a test to check if we import 3rd party library when we do `import pyspark` to prevent similar issues in the future.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    `test_shuffle` passed locally, the rest is on CI.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No.
    
    Closes #54897 from gaogaotiantian/lazy-import-psutil.
    
    Authored-by: Tian Gao <ga...@hotmail.com>
    Signed-off-by: Ruifeng Zheng <ru...@apache.org>
---
 python/pyspark/shuffle.py | 21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)

diff --git a/python/pyspark/shuffle.py b/python/pyspark/shuffle.py
index 30147afe0086..dc43353e36db 100644
--- a/python/pyspark/shuffle.py
+++ b/python/pyspark/shuffle.py
@@ -35,13 +35,19 @@ from pyspark.serializers import (
 )
 from pyspark.util import fail_on_stopiteration
 
-try:
-    import psutil
+process = None
 
-    process = None
 
-    def get_used_memory():
-        """Return the used memory in MiB"""
+def get_used_memory():
+    """Return the used memory in MiB"""
+    try:
+        import psutil
+
+        has_psutil = True
+    except ImportError:
+        has_psutil = False
+
+    if has_psutil:
         global process
         if process is None or process._pid != os.getpid():
             process = psutil.Process(os.getpid())
@@ -51,10 +57,7 @@ try:
             info = process.get_memory_info()
         return info.rss >> 20
 
-except ImportError:
-
-    def get_used_memory():
-        """Return the used memory in MiB"""
+    else:
         if platform.system() == "Linux":
             for line in open("/proc/self/status"):
                 if line.startswith("VmRSS:"):


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org