You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Boyang Jerry Peng (Jira)" <ji...@apache.org> on 2022/06/24 21:02:00 UTC

[jira] [Created] (SPARK-39593) Configurable State Checkpointing Frequency in Structured Streaming

Boyang Jerry Peng created SPARK-39593:
-----------------------------------------

             Summary: Configurable State Checkpointing Frequency in Structured Streaming
                 Key: SPARK-39593
                 URL: https://issues.apache.org/jira/browse/SPARK-39593
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 3.3.0
            Reporter: Boyang Jerry Peng


Currently, for stateful pipelines state changes are checkpointed for every micro-batch. State checkpoints can contribute significantly to the latency of a micro-batch.  If state is checkpointed less frequently, its effect on batch latency can be amortized.  This can be used in conjunction with asynchronous state checkpointing to further reduce the cost in latency state checkpointing may incur.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org