You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "jiateoh (via GitHub)" <gi...@apache.org> on 2026/03/27 19:36:51 UTC

Re: [PR] [WIP][SPARK-56248][PYTHON][SS] Optimize python TWS stateful processor serialization calls [spark]

jiateoh commented on code in PR #55039:
URL: https://github.com/apache/spark/pull/55039#discussion_r3002842788


##########
python/pyspark/sql/streaming/stateful_processor_api_client.py:
##########
@@ -27,12 +27,45 @@
     Row,
 )
 from pyspark.sql.pandas.types import convert_pandas_using_numpy_type
+from pyspark.sql.utils import has_numpy
 from pyspark.serializers import CPickleSerializer
 from pyspark.errors import PySparkRuntimeError
 import uuid
 
 __all__ = ["StatefulProcessorApiClient", "StatefulProcessorHandleState"]
 
+if has_numpy:
+    import numpy as np
+
+    def _normalize_value(v: Any) -> Any:
+        # Convert NumPy types to Python primitive types.
+        if isinstance(v, np.generic):
+            return v.tolist()
+        # Named tuples (collections.namedtuple or typing.NamedTuple) and Row both
+        # require positional arguments and cannot be instantiated
+        # with a generator expression.
+        if isinstance(v, Row) or (isinstance(v, tuple) and hasattr(v, "_fields")):
+            return type(v)(*[_normalize_value(e) for e in v])
+        # List / tuple: recursively normalize each element
+        if isinstance(v, (list, tuple)):
+            return type(v)(_normalize_value(e) for e in v)
+        # Dict: normalize both keys and values
+        if isinstance(v, dict):
+            return {_normalize_value(k): _normalize_value(val) for k, val in v.items()}
+        # Address a couple of pandas dtypes too.
+        elif hasattr(v, "to_pytimedelta"):
+            return v.to_pytimedelta()
+        elif hasattr(v, "to_pydatetime"):
+            return v.to_pydatetime()
+        return v
+
+    def _normalize_tuple(data: Tuple) -> Tuple:
+        return tuple(_normalize_value(v) for v in data)

Review Comment:
   Another fastpath option being considered: prevalidate the schema, if it's all non-nested we can skip some checks. Prototype below but still investigating:
   
   ```def _is_simple_schema(schema: StructType) -> bool:
       """True if schema has no nested structs, arrays, or maps."""
       return all(
           not isinstance(f.dataType, (StructType, ArrayType, MapType))
           for f in schema.fields
       )
   
      def _normalize_tuple_simple(data: Tuple) -> Tuple:
           """Fast path: 1 isinstance per element instead of 7 (no nested types)."""
           return tuple(v.tolist() if isinstance(v, np.generic) else v for v in data)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org