[jira] [Resolved] (SPARK-56369) Introduce rangeScan API into StateStore

["Jungtaek Lim (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-56369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jungtaek Lim resolved SPARK-56369.
----------------------------------
    Fix Version/s: 4.2.0
       Resolution: Fixed

Issue resolved by pull request 55226
[https://github.com/apache/spark/pull/55226]

> Introduce rangeScan API into StateStore
> ---------------------------------------
>
>                 Key: SPARK-56369
>                 URL: https://issues.apache.org/jira/browse/SPARK-56369
>             Project: Spark
>          Issue Type: Task
>          Components: Structured Streaming
>    Affects Versions: 4.2.0
>            Reporter: Jungtaek Lim
>            Assignee: Jungtaek Lim
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.2.0
>
>
> Currently, iterator API in StateStore only receives column family which scopes the range of iteration to the specific column family. It still seeks for the very beginning of the key for column family, scans through all tombstones internally during scanning for keys till either the iterator has fully consumed or iterator is closed explicitly.
> Recently we made some improvement on prefixScan to scope lower bound and upper bound, based on the characteristic of prefixScan. The improvement is not only useful for prefixScan but also useful for any sort of range scan.
> We want to introduce a new API in StateStore to provide the iterator with accepting the boundary of the keys. With this approach, we can skip tombstones very efficiently in both lower boundary side and upper boundary side.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org