You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Chaoqin Li (Jira)" <ji...@apache.org> on 2023/05/09 18:20:00 UTC
[jira] [Commented] (SPARK-43421) Implement changelog checkpointing for RocksDB state store
[ https://issues.apache.org/jira/browse/SPARK-43421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721037#comment-17721037 ]
Chaoqin Li commented on SPARK-43421:
------------------------------------
[#Design doc]https://docs.google.com/document/d/1c7EkHkguhE7WiIWAKMkvDi3kjd1V98oRtdjk6L_iHoA
> Implement changelog checkpointing for RocksDB state store
> ---------------------------------------------------------
>
> Key: SPARK-43421
> URL: https://issues.apache.org/jira/browse/SPARK-43421
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 3.4.0
> Reporter: Chaoqin Li
> Priority: Major
>
> We have identified state checkpointing latency as one of the major performance bottlenecks for stateful streaming queries. Currently, RocksDB state store pauses the RocksDB instances to upload a snapshot to the cloud when committing a batch, which is heavy weight and has unpredictable performance.
> In order to reduce the checkpoint duration and end to end latency, we propose to
> 1. During state commit, make the state of a microbatch durable by syncing the changelog instead of the state snapshot to the checkpoint directory.
> 2. Upload snapshot in the background to enable changelog purging and faster failure recovery.
> In this way, we allow the RocksDB instance to run uninterruptibly, which improves RocksDB operation performance. This also dramatically reduces the commit time and batch duration because we are uploading a smaller amount of data during state commit. With this change, stateful query with RocksDB state store will have lower and more predictable latency.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org