You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ka...@apache.org on 2025/04/24 05:46:30 UTC

(spark) branch branch-4.0 updated: [SPARK-51889][PYTHON][SS] Fix a bug for MapState clear() in Python TWS

This is an automated email from the ASF dual-hosted git repository.

kabhwan pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.0 by this push:
     new 7305cef149cb [SPARK-51889][PYTHON][SS] Fix a bug for MapState clear() in Python TWS
7305cef149cb is described below

commit 7305cef149cba182019462848679e26100fc29cc
Author: bogao007 <bo...@databricks.com>
AuthorDate: Thu Apr 24 14:43:43 2025 +0900

    [SPARK-51889][PYTHON][SS] Fix a bug for MapState clear() in Python TWS
    
    Fix a bug for MapState clear() in Python TWS
    
    Without this change, MapState clear() would not work in Python.
    
    No
    
    Added unit test
    
    No
    
    Closes #50686 from bogao007/mapstate-clear.
    
    Authored-by: bogao007 <bo...@databricks.com>
    Signed-off-by: Jungtaek Lim <ka...@gmail.com>
    (cherry picked from commit 6eb966812b328a29ccc6ac1531d0a4258e6c90c7)
    Signed-off-by: Jungtaek Lim <ka...@gmail.com>
---
 python/pyspark/sql/tests/pandas/test_pandas_transform_with_state.py    | 3 +++
 .../python/streaming/TransformWithStateInPandasStateServer.scala       | 3 +++
 2 files changed, 6 insertions(+)

diff --git a/python/pyspark/sql/tests/pandas/test_pandas_transform_with_state.py b/python/pyspark/sql/tests/pandas/test_pandas_transform_with_state.py
index d5c27806197f..0991157a6752 100644
--- a/python/pyspark/sql/tests/pandas/test_pandas_transform_with_state.py
+++ b/python/pyspark/sql/tests/pandas/test_pandas_transform_with_state.py
@@ -1965,6 +1965,9 @@ class MapStateProcessor(StatefulProcessor):
         assert next(map_iter)[1] == (value2,)
         self.map_state.removeKey(key1)
         assert not self.map_state.containsKey(key1)
+        assert self.map_state.exists()
+        self.map_state.clear()
+        assert not self.map_state.exists()
         yield pd.DataFrame({"id": key, "countAsString": str(count)})
 
     def close(self) -> None:
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPandasStateServer.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPandasStateServer.scala
index f13df63c8f26..9a1dcf9438ea 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPandasStateServer.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPandasStateServer.scala
@@ -624,6 +624,9 @@ class TransformWithStateInPandasStateServer(
           mapStateInfo.keyDeserializer)
         mapStateInfo.mapState.removeKey(keyRow)
         sendResponse(0)
+      case MapStateCall.MethodCase.CLEAR =>
+        mapStateInfo.mapState.clear()
+        sendResponse(0)
       case _ =>
         throw new IllegalArgumentException("Invalid method call")
     }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org