[jira] [Commented] (SPARK-39134) Add custom metric of skipped null values for stream join operator

["Apache Spark (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-39134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534092#comment-17534092 ] 

Apache Spark commented on SPARK-39134:
--------------------------------------

User 'viirya' has created a pull request for this issue:
https://github.com/apache/spark/pull/36491

> Add custom metric of skipped null values for stream join operator
> -----------------------------------------------------------------
>
>                 Key: SPARK-39134
>                 URL: https://issues.apache.org/jira/browse/SPARK-39134
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 3.4.0
>            Reporter: L. C. Hsieh
>            Priority: Major
>
> When STATE_STORE_SKIP_NULLS_FOR_STREAM_STREAM_JOINS is enabled, Spark will skip null values in hash-based stream-stream joins. But currently we cannot track if this is happened and how many it is. For our customers, they have an interest to know the number of values skipped for better tracking purpose.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org