You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2025/04/26 00:10:00 UTC

[jira] [Assigned] (SPARK-51904) Removing async metadata purging for OperatorStateMetadataV2

     [ https://issues.apache.org/jira/browse/SPARK-51904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jungtaek Lim reassigned SPARK-51904:
------------------------------------

    Assignee: Eric Marnadi

> Removing async metadata purging for OperatorStateMetadataV2
> -----------------------------------------------------------
>
>                 Key: SPARK-51904
>                 URL: https://issues.apache.org/jira/browse/SPARK-51904
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 4.0.0, 4.1.0
>            Reporter: Eric Marnadi
>            Assignee: Eric Marnadi
>            Priority: Major
>              Labels: pull-request-available
>
> Currently, we don't want to purge OperatorStateMetadataV2 or StateSchemaV3 files, so we need to remove the relevant call from MicrobatchExecution. The changes are needed because we cannot purge schema files because these are necessary until full rewrite is implemented.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org