You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Huanli Wang (Jira)" <ji...@apache.org> on 2024/12/21 04:08:00 UTC

[jira] [Created] (SPARK-50642) Fix the state schema for FlatMapGroupsWithState in spark connect when there is no initial state

Huanli Wang created SPARK-50642:
-----------------------------------

             Summary: Fix the state schema for FlatMapGroupsWithState in spark connect when there is no initial state
                 Key: SPARK-50642
                 URL: https://issues.apache.org/jira/browse/SPARK-50642
             Project: Spark
          Issue Type: Improvement
          Components: Connect, Structured Streaming
    Affects Versions: 3.5.0
            Reporter: Huanli Wang
             Fix For: 4.0.0


In spark connect, when there is no initial state, we derived the state schema from the input:
 * create the initialDs from the original input: [https://github.com/apache/spark/blob/master/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala#L679-L689]
 * derived the state expression encoder from this `initialDs` which is incorrect: [https://github.com/apache/spark/blob/master/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala#L729]

 

Our unit tests fail to cover this case because it doesn't do the state update: [https://github.com/apache/spark/blob/master/connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/streaming/FlatMapGroupsWithStateStreamingSuite.scala#L55-L59]

after changing the `stateFunc` to the following
{code:java}
val stateFunc =
  (key: String, values: Iterator[ClickEvent], state: GroupState[ClickState]) => {
    if (state.exists) throw new IllegalArgumentException("state.exists should be false")
    val newState = ClickState(key, values.size)
    state.update(newState)
    Iterator(newState)
  } {code}
the test is actually failing with



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org