You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Chaoqin Li (Jira)" <ji...@apache.org> on 2023/10/16 17:27:00 UTC
[jira] [Updated] (SPARK-45558) Introduce a metadata file for streaming stateful operator
[ https://issues.apache.org/jira/browse/SPARK-45558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chaoqin Li updated SPARK-45558:
-------------------------------
Description:
The information to store in the metadata file:
* operator name (no need to be unique among stateful operators in the query)
* state store name
* numColumnsPrefixKey: > 0 if prefix scan is enabled, 0 otherwise
The body of metadata file will be in json format. The metadata file will be versioned just as other streaming metadata file to be future proof.
The metadata file will improve expose more information about the state store, improves debugability and facilitate the development of state related feature such as reading and writing state and state repartitioning.
was:
The information to store in the metadata file:
* operator name (no need to be unique among stateful operators in the query)
* state store name
* numColumnsPrefixKey: > 0 if prefix scan is enabled, 0 otherwise
The body of metadata file will be in json format. The metadata file will be versioned just as other streaming metadata file to be future proof.
> Introduce a metadata file for streaming stateful operator
> ---------------------------------------------------------
>
> Key: SPARK-45558
> URL: https://issues.apache.org/jira/browse/SPARK-45558
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 3.5.0
> Reporter: Chaoqin Li
> Priority: Major
>
> The information to store in the metadata file:
> * operator name (no need to be unique among stateful operators in the query)
> * state store name
> * numColumnsPrefixKey: > 0 if prefix scan is enabled, 0 otherwise
> The body of metadata file will be in json format. The metadata file will be versioned just as other streaming metadata file to be future proof.
> The metadata file will improve expose more information about the state store, improves debugability and facilitate the development of state related feature such as reading and writing state and state repartitioning.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org