You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2026/07/09 13:28:17 UTC

[PR] [DO-NOT-MERGE] Do not import numpy at 'import pyspark' time in stateful_processor_api_client [spark]

HyukjinKwon opened a new pull request, #57160:
URL: https://github.com/apache/spark/pull/57160

   ### What changes were proposed in this pull request?
   
   This PR makes `pyspark/sql/streaming/stateful_processor_api_client.py` resolve NumPy lazily instead of importing it at module load time.
   
   SPARK-57760 moved `import numpy as np` to the module top level (inside a `try/except ImportError` block). This module is transitively imported by `import pyspark` (`pyspark` -> `pyspark.sql` -> `pyspark.sql.streaming` -> `stateful_processor_api_client`), so NumPy is now imported during `import pyspark`.
   
   Instead of the top-level import, this PR:
   - adds a cached `has_numpy()` helper that performs the availability check on first use, and
   - imports `numpy` locally inside `_normalize_state_value` (only invoked when normalizing state values).
   
   The observable behavior is unchanged.
   
   ### Why are the changes needed?
   
   `import pyspark` is expected to stay fast and free of 3rd party libraries. `python/pyspark/tests/test_import_spark.py::ImportSparkTest::test_import_spark_libraries` enforces this and is currently failing on `master` / `branch-4.x`:
   
   ```
   AssertionError: Unexpected 3rd party package 'numpy' imported during 'import pyspark'
   ```
   
   This has been failing the scheduled Non-ANSI builds (e.g. `build_non_ansi.yml` on `branch-4.x`).
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   - `python -X importtime -c "import pyspark"` no longer shows `numpy` in the import trace.
   - `python/pyspark/tests/test_import_spark.py` passes.
   - Verified `_normalize_state_value` still converts NumPy scalars and nested containers correctly, and the scalar fast-path is unchanged.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [DO-NOT-MERGE] Do not import numpy at 'import pyspark' time in stateful_processor_api_client [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #57160: [DO-NOT-MERGE] Do not import numpy at 'import pyspark' time in stateful_processor_api_client
URL: https://github.com/apache/spark/pull/57160


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58071][SS][PYTHON] Do not import numpy at 'import pyspark' time in stateful_processor_api_client [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #57160:
URL: https://github.com/apache/spark/pull/57160#issuecomment-4929872970

   Superseded by #57169 (same branch; this PR was auto-closed and could not be reopened after the branch head was updated).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org