You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2026/01/09 02:51:00 UTC

[jira] [Assigned] (SPARK-54122) Improve the testing experience for TransformWithState

     [ https://issues.apache.org/jira/browse/SPARK-54122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jungtaek Lim reassigned SPARK-54122:
------------------------------------

    Assignee: Dmytro Fedoriaka

> Improve the testing experience for TransformWithState
> -----------------------------------------------------
>
>                 Key: SPARK-54122
>                 URL: https://issues.apache.org/jira/browse/SPARK-54122
>             Project: Spark
>          Issue Type: New Feature
>          Components: Structured Streaming
>    Affects Versions: 4.1.0
>            Reporter: Dmytro Fedoriaka
>            Assignee: Dmytro Fedoriaka
>            Priority: Minor
>              Labels: pull-request-available
>
> Currently it's hard to write unit tests for user's implementation of StatefulProcessor used in TransformWithState. User either needs to test it by running actual streaming query, or they need to refactor business logic into a separate function (that will be called from handleInputRows), and write unit tests for that function.
>  
> I propose to add a simple testing framework for TransformWithState that will allow users easily test their business logic inside StatefulProcessor.
>  
> On high-level, it's a class TwsTester that takes StatefulProcessor and allows to feed in input rows and immediately returns what rows would be produced by TWS. It also allows to set and inspect state. This can be used in unit tests without having to run streaming query and it won't need RocksDB (it will use in-memory state store). I will start with implementing this for Scala users, potentially making it available for Python users later.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org