[jira] [Commented] (SPARK-53069) Fix incorrect metrics with virtual column families

["Anish Shrigondekar (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-53069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18012189#comment-18012189 ] 

Anish Shrigondekar commented on SPARK-53069:
--------------------------------------------

PR merged here -> https://github.com/apache/spark/pull/51790

> Fix incorrect metrics with virtual column families
> --------------------------------------------------
>
>                 Key: SPARK-53069
>                 URL: https://issues.apache.org/jira/browse/SPARK-53069
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 4.1.0
>            Reporter: Dylan Wong
>            Priority: Major
>              Labels: pull-request-available
>
> There is a state store incorrect metric issue related to the variables numInternalKeysOnLoadedVersion and numInternalKeysOnWritingVersion not being updated correctly in memory. This caused them to fall out of sync with the actual key count on disk, as determined by countKeys(), leading to incorrect metrics being emitted.
>  
> The root cause of this issue is that the replayChangelog function is calling put/remove/merge with the default column family, rather than the correct column family for each key which causes the wrong counter (numInternalKeysOnWritingVersion vs numKeysOnWritingVersion) to be incremented.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org