[jira] [Updated] (SPARK-58072) Avoid eagerly importing numpy in StatefulProcessorApiClient

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-58072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-58072:
-----------------------------------
    Labels: pull-request-available  (was: )

> Avoid eagerly importing numpy in StatefulProcessorApiClient
> -----------------------------------------------------------
>
>                 Key: SPARK-58072
>                 URL: https://issues.apache.org/jira/browse/SPARK-58072
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark, Structured Streaming, Tests
>    Affects Versions: 4.1.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> SPARK-57760 hoisted `import numpy as np` to module top-level in `python/pyspark/sql/streaming/stateful_processor_api_client.py` (to define `has_numpy` and `_normalize_state_value` once). However, that module is on the eager `import pyspark` import chain:
> {code}
> pyspark -> pyspark.sql -> pyspark.sql.context -> pyspark.sql.session
>         -> pyspark.sql.dataframe -> pyspark.sql.streaming
>         -> pyspark.sql.streaming.stateful_processor
>         -> pyspark.sql.streaming.stateful_processor_api_client -> import numpy
> {code}
> As a result, `import pyspark` now eagerly imports numpy, which breaks `pyspark.tests.test_import_spark.ImportSparkTest.test_import_spark_libraries` (added in SPARK-56242). That test asserts no 3rd party package is imported during `import pyspark` to keep import fast:
> {code}
> AssertionError: Unexpected 3rd party package 'numpy' imported during 'import pyspark'
> {code}
> This fails several scheduled CI jobs that run the test with numpy installed, e.g.:
> - Build / Java25 (Scala 2.13, JDK 25) on master
> - Build / RocksDB as UI Backend (JDK 17) on master
> - Build / Python-only (Minimum dependencies of PySpark) on master
> - Build / Java21 (Scala 2.13, JDK 21) on branch-4.x
> The offending commit is present on both master and branch-4.x, so the fix should be backported to branch-4.x as well.
> Fix: resolve `has_numpy` via `importlib.util.find_spec("numpy")` (detects numpy without importing it) and import numpy lazily inside `_normalize_state_value`. This restores the pre-SPARK-57760 lazy-import behavior while keeping the benefit of defining `_normalize_state_value` once.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org