You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/05/25 08:01:00 UTC
[jira] [Updated] (SPARK-57052) [SS] Add state row format validation to multiGet and valuesIterator in RocksDBStateStoreProvider
[ https://issues.apache.org/jira/browse/SPARK-57052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated SPARK-57052:
-----------------------------------
Labels: pull-request-available (was: )
> [SS] Add state row format validation to multiGet and valuesIterator in RocksDBStateStoreProvider
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-57052
> URL: https://issues.apache.org/jira/browse/SPARK-57052
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 4.0.0
> Reporter: Anupam Yadav
> Priority: Major
> Labels: pull-request-available
>
> RocksDBStateStoreProvider performs state row format validation (via StateStoreProvider.validateStateRowFormat) on most read-path methods to catch schema incompatibilities early. However, two read methods that decode rows are missing this validation:
> ||Method||Decodes rows?||Has validation?||
> |get()|Yes|(/)|
> |iterator()|Yes|(/)|
> |prefixScan()|Yes|(/)|
> |rangeScan()|Yes|(/)|
> |multiGet()|Yes|(x)|
> |valuesIterator()|Yes|(x)|
> multiGet() decodes values via encodedValues.map(kvEncoder._2.decodeValue) but never calls validateStateRowFormat. Similarly, valuesIterator() decodes values via valueEncoder.decodeValues() without validation.
> If a stateful operator evolves its schema between restarts, these methods will silently return corrupted data instead of failing fast with StateStoreKeyRowFormatValidationFailure / StateStoreValueRowFormatValidationFailure.
> h3. Fix
> Add the same validateStateRowFormat guard (gated by !isValidated && value != null && !useColumnFamilies) to multiGet() and valuesIterator(), matching the existing pattern in get() and iterator().
> h3. Acceptance Criteria
> * multiGet() calls validateStateRowFormat on the first non-null result, consistent with get()
> * valuesIterator() calls validateStateRowFormat on the first decoded row, consistent with iterator()
> * Add unit tests that verify StateStoreKeyRowFormatValidationFailure is thrown from both methods when a mismatched schema is used
> * Existing RocksDBStateStoreProviderSuite tests pass with no regressions
> ----
> *Prior art:* SPARK-56539 / [PR #55468|https://github.com/apache/spark/pull/55468] added the same validation to prefixScan and rangeScan.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org