You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ka...@apache.org on 2025/04/24 05:46:30 UTC
(spark) branch branch-4.0 updated: [SPARK-51889][PYTHON][SS] Fix a bug for MapState clear() in Python TWS
This is an automated email from the ASF dual-hosted git repository.
kabhwan pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 7305cef149cb [SPARK-51889][PYTHON][SS] Fix a bug for MapState clear() in Python TWS
7305cef149cb is described below
commit 7305cef149cba182019462848679e26100fc29cc
Author: bogao007 <bo...@databricks.com>
AuthorDate: Thu Apr 24 14:43:43 2025 +0900
[SPARK-51889][PYTHON][SS] Fix a bug for MapState clear() in Python TWS
Fix a bug for MapState clear() in Python TWS
Without this change, MapState clear() would not work in Python.
No
Added unit test
No
Closes #50686 from bogao007/mapstate-clear.
Authored-by: bogao007 <bo...@databricks.com>
Signed-off-by: Jungtaek Lim <ka...@gmail.com>
(cherry picked from commit 6eb966812b328a29ccc6ac1531d0a4258e6c90c7)
Signed-off-by: Jungtaek Lim <ka...@gmail.com>
---
python/pyspark/sql/tests/pandas/test_pandas_transform_with_state.py | 3 +++
.../python/streaming/TransformWithStateInPandasStateServer.scala | 3 +++
2 files changed, 6 insertions(+)
diff --git a/python/pyspark/sql/tests/pandas/test_pandas_transform_with_state.py b/python/pyspark/sql/tests/pandas/test_pandas_transform_with_state.py
index d5c27806197f..0991157a6752 100644
--- a/python/pyspark/sql/tests/pandas/test_pandas_transform_with_state.py
+++ b/python/pyspark/sql/tests/pandas/test_pandas_transform_with_state.py
@@ -1965,6 +1965,9 @@ class MapStateProcessor(StatefulProcessor):
assert next(map_iter)[1] == (value2,)
self.map_state.removeKey(key1)
assert not self.map_state.containsKey(key1)
+ assert self.map_state.exists()
+ self.map_state.clear()
+ assert not self.map_state.exists()
yield pd.DataFrame({"id": key, "countAsString": str(count)})
def close(self) -> None:
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPandasStateServer.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPandasStateServer.scala
index f13df63c8f26..9a1dcf9438ea 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPandasStateServer.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPandasStateServer.scala
@@ -624,6 +624,9 @@ class TransformWithStateInPandasStateServer(
mapStateInfo.keyDeserializer)
mapStateInfo.mapState.removeKey(keyRow)
sendResponse(0)
+ case MapStateCall.MethodCase.CLEAR =>
+ mapStateInfo.mapState.clear()
+ sendResponse(0)
case _ =>
throw new IllegalArgumentException("Invalid method call")
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org