[jira] [Created] (SPARK-58073) Avoid eagerly importing numpy in StatefulProcessorApiClient

["Hyukjin Kwon (Jira)" <ji...@apache.org>]

Hyukjin Kwon created SPARK-58073:
------------------------------------

             Summary: Avoid eagerly importing numpy in StatefulProcessorApiClient
                 Key: SPARK-58073
                 URL: https://issues.apache.org/jira/browse/SPARK-58073
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming, PySpark
    Affects Versions: 4.2.0, 5.0.0
            Reporter: Hyukjin Kwon


SPARK-57760 hoisted {{import numpy as np}} to module level in {{python/pyspark/sql/streaming/stateful_processor_api_client.py}} (to define {{has_numpy}} and {{_normalize_state_value}} once). That module is on the eager {{import pyspark}} chain (pyspark -> sql -> context -> session -> dataframe -> streaming -> stateful_processor -> stateful_processor_api_client), so it now imports numpy during {{import pyspark}}.

This breaks {{pyspark.tests.test_import_spark}} (added in SPARK-56242), which asserts that no 3rd party package is imported during {{import pyspark}}:

{code}
AssertionError: Unexpected 3rd party package 'numpy' imported during 'import pyspark'
{code}

As a result, the following scheduled jobs fail:
* {{Build / Python-only (Minimum dependencies of PySpark)}} on master
* {{Build / Java21 (Scala 2.13, JDK 21)}} on branch-4.x

The offending commit is present on both master (0af85eee8171) and branch-4.x (181acbd8db57), so the fix needs to be backported to branch-4.x.

Fix: resolve {{has_numpy}} via {{importlib.util.find_spec("numpy")}} (which does not import numpy) and import numpy lazily inside {{_normalize_state_value}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org