You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2026/10/13 04:46:00 UTC
[jira] [Resolved] (SPARK-60181) Guard stream-stream join state format and RocksDB merge operator compatibility
[ https://issues.apache.org/jira/browse/SPARK-60181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim resolved SPARK-60181.
----------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
Issue resolved by pull request 59389
[https://github.com/apache/spark/pull/59389]
> Guard stream-stream join state format and RocksDB merge operator compatibility
> ------------------------------------------------------------------------------
>
> Key: SPARK-60181
> URL: https://issues.apache.org/jira/browse/SPARK-60181
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 4.3.0, 4.4.0
> Reporter: Jungtaek Lim
> Assignee: Jungtaek Lim
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> Recent stream-stream state format versions require RocksDB state store provider to be available, but the default state store provider is HDFS so it could be possible that both configs are not set for the same.
> In addition, stream-stream state format version 4 requires RocksDB's merge operator version to 2, which can happen the same.
> In both cases, it is better for the engine to adjust the configs to make the query run rather than leaving the join operator to be broken.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org