You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Huanli Wang (Jira)" <ji...@apache.org> on 2024/12/21 04:08:00 UTC
[jira] [Created] (SPARK-50642) Fix the state schema for FlatMapGroupsWithState in spark connect when there is no initial state
Huanli Wang created SPARK-50642:
-----------------------------------
Summary: Fix the state schema for FlatMapGroupsWithState in spark connect when there is no initial state
Key: SPARK-50642
URL: https://issues.apache.org/jira/browse/SPARK-50642
Project: Spark
Issue Type: Improvement
Components: Connect, Structured Streaming
Affects Versions: 3.5.0
Reporter: Huanli Wang
Fix For: 4.0.0
In spark connect, when there is no initial state, we derived the state schema from the input:
* create the initialDs from the original input: [https://github.com/apache/spark/blob/master/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala#L679-L689]
* derived the state expression encoder from this `initialDs` which is incorrect: [https://github.com/apache/spark/blob/master/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala#L729]
Our unit tests fail to cover this case because it doesn't do the state update: [https://github.com/apache/spark/blob/master/connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/streaming/FlatMapGroupsWithStateStreamingSuite.scala#L55-L59]
after changing the `stateFunc` to the following
{code:java}
val stateFunc =
(key: String, values: Iterator[ClickEvent], state: GroupState[ClickState]) => {
if (state.exists) throw new IllegalArgumentException("state.exists should be false")
val newState = ClickState(key, values.size)
state.update(newState)
Iterator(newState)
} {code}
the test is actually failing with
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org