You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Elvinas Pilypas (Jira)" <ji...@apache.org> on 2025/10/15 11:52:00 UTC

[jira] [Created] (SPARK-53918) asyncCheckpoint.enabled incompatible with transformWithStateInPandas (StatefulProcessor)

Elvinas Pilypas created SPARK-53918:
---------------------------------------

             Summary: asyncCheckpoint.enabled incompatible with transformWithStateInPandas (StatefulProcessor)
                 Key: SPARK-53918
                 URL: https://issues.apache.org/jira/browse/SPARK-53918
             Project: Spark
          Issue Type: Bug
          Components: PySpark, Structured Streaming
    Affects Versions: 4.0.1, 4.0.0
         Environment: Databricks standard workspace

Databricks runtime: 17.0 (Scala 2.13, Spark 4.0.0)

Node type: Standard_D4ads_v5
            Reporter: Elvinas Pilypas


In structured streaming job, we use transformWithStateInPandas stateful processing and we have com.databricks.sql.streaming.state.RocksDBStateStoreProvider enabled. Everything works as expected.

But when we enable:
spark.conf.set("spark.databricks.streaming.statefulOperator.asyncCheckpoint.enabled", "true")

We get an error: org.apache.spark.SparkUnsupportedOperationException: Synchronous commit is not supported. Use asynchronous commit.
 
We have successfully replicated that in notebook, and also tried the same thing with applyInPandasWithState. * transformWithStateInPandas - throws Synchronous commit error
 * applyInPandasWithState - works as expected

We followed this guide: [https://learn.microsoft.com/en-us/azure/databricks/structured-streaming/async-checkpointing] and also used chatGPT for references.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org