(spark) branch branch-4.x updated: [SPARK-59013][PYTHON] Remove redundant Python 3 boilerplate in PySpark serializers

[yi...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

Yicong-Huang pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new b820e79644cc [SPARK-59013][PYTHON] Remove redundant Python 3 boilerplate in PySpark serializers
b820e79644cc is described below

commit b820e79644ccf487233b5128134a513d7e75de67
Author: Yicong Huang <17...@users.noreply.github.com>
AuthorDate: Thu Aug 27 00:31:08 2026 +0000

    [SPARK-59013][PYTHON] Remove redundant Python 3 boilerplate in PySpark serializers
    
    ### What changes were proposed in this pull request?
    
    This PR removes redundant boilerplate in `python/pyspark/serializers.py` that is either a Python 2 leftover or made unnecessary by Python 3 semantics. None of it changes behavior:
    
    - Drop `FramedSerializer.dumps`, a verbatim duplicate (signature and docstring) of the inherited abstract `Serializer.dumps`.
    - Drop `Serializer.__ne__`: Python 3 derives `!=` from `__eq__` automatically. `__hash__` is intentionally kept, since it is not auto-derived once `__eq__` is defined.
    - Drop the no-op `FramedSerializer.__init__(self)` call in `CompressedSerializer.__init__`: neither `Serializer` nor `FramedSerializer` defines `__init__`, so it only resolves to `object.__init__`.
    - Collapse the duplicate `itertools` import: remove `from itertools import chain, product` and qualify the four usages as `itertools.chain` / `itertools.product`, consistent with the existing `itertools.islice` usage in the same file.
    
    ### Why are the changes needed?
    
    These are dead or redundant lines that add noise without adding behavior. Removing them makes the module easier to read and maintain.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Existing `python/pyspark/tests/test_serializers.py` passes. The change is behavior-preserving: the removed `dumps` override and `__init__` call were no-ops, and `!=` continues to work via Python 3's automatic derivation from `__eq__`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No.
    
    Closes #58297 from Yicong-Huang/cleanup-serializers-boilerplate.
    
    Authored-by: Yicong Huang <17...@users.noreply.github.com>
    Signed-off-by: Yicong-Huang <17...@users.noreply.github.com>
    (cherry picked from commit e62838cdebf59a2540e8d0c581408828a32cb306)
    Signed-off-by: Yicong-Huang <17...@users.noreply.github.com>
---
 python/pyspark/serializers.py | 20 ++++----------------
 1 file changed, 4 insertions(+), 16 deletions(-)

diff --git a/python/pyspark/serializers.py b/python/pyspark/serializers.py
index 9f72e771b68e..8b9cea297906 100644
--- a/python/pyspark/serializers.py
+++ b/python/pyspark/serializers.py
@@ -63,7 +63,6 @@ import struct
 import sys
 import types
 import zlib
-from itertools import chain, product
 
 pickle_protocol = pickle.HIGHEST_PROTOCOL
 
@@ -125,9 +124,6 @@ class Serializer:
     def __eq__(self, other):
         return isinstance(other, self.__class__) and self.__dict__ == other.__dict__
 
-    def __ne__(self, other):
-        return not self.__eq__(other)
-
     def __repr__(self):
         return "%s()" % self.__class__.__name__
 
@@ -172,13 +168,6 @@ class FramedSerializer(Serializer):
             raise EOFError
         return self.loads(obj)
 
-    def dumps(self, obj):
-        """
-        Serialize an object into a byte array.
-        When batching is used, this will be called with an array of objects.
-        """
-        raise NotImplementedError
-
     def loads(self, obj):
         """
         Deserialize an object from a byte array.
@@ -211,7 +200,7 @@ class BatchedSerializer(Serializer):
         self.serializer.dump_stream(self._batched(iterator), stream)
 
     def load_stream(self, stream):
-        return chain.from_iterable(self._load_stream_without_unbatching(stream))
+        return itertools.chain.from_iterable(self._load_stream_without_unbatching(stream))
 
     def _load_stream_without_unbatching(self, stream):
         return self.serializer.load_stream(stream)
@@ -290,10 +279,10 @@ class CartesianDeserializer(Serializer):
         val_batch_stream = self.val_ser._load_stream_without_unbatching(stream)
         for key_batch, val_batch in zip(key_batch_stream, val_batch_stream):
             # for correctness with repeated cartesian/zip this must be returned as one batch
-            yield product(key_batch, val_batch)
+            yield itertools.product(key_batch, val_batch)
 
     def load_stream(self, stream):
-        return chain.from_iterable(self._load_stream_without_unbatching(stream))
+        return itertools.chain.from_iterable(self._load_stream_without_unbatching(stream))
 
     def __repr__(self):
         return "CartesianDeserializer(%s, %s)" % (str(self.key_ser), str(self.val_ser))
@@ -327,7 +316,7 @@ class PairDeserializer(Serializer):
             yield zip(key_batch, val_batch)
 
     def load_stream(self, stream):
-        return chain.from_iterable(self._load_stream_without_unbatching(stream))
+        return itertools.chain.from_iterable(self._load_stream_without_unbatching(stream))
 
     def __repr__(self):
         return "PairDeserializer(%s, %s)" % (str(self.key_ser), str(self.val_ser))
@@ -484,7 +473,6 @@ class CompressedSerializer(FramedSerializer):
     """
 
     def __init__(self, serializer):
-        FramedSerializer.__init__(self)
         assert isinstance(serializer, FramedSerializer), "serializer must be a FramedSerializer"
         self.serializer = serializer
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org