[jira] [Commented] (SPARK-59986) Expose trigger metadata in StreamingQueryProgress JSON

["Alyssa Gorbaneva (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-59986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18123582#comment-18123582 ] 

Alyssa Gorbaneva commented on SPARK-59986:
------------------------------------------

Could a Spark committer assign this issue to me? I am the reporter and author of apache/spark#59247 and am working on the implementation.

> Expose trigger metadata in StreamingQueryProgress JSON
> ------------------------------------------------------
>
>                 Key: SPARK-59986
>                 URL: https://issues.apache.org/jira/browse/SPARK-59986
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 5.0.0
>            Reporter: Alyssa Gorbaneva
>            Priority: Major
>              Labels: pull-request-available
>
> h2. Motivation
> StreamingQueryProgress JSON reports query identifiers, batch timings, source, sink, state operator, and observed metrics, but it does not report the Trigger selected for the query. Consumers of progress events cannot determine whether a query uses ProcessingTime, Continuous, RealTime, AvailableNow, or OneTime from the progress payload.
> h2. Proposed change
> Add an optional top-level trigger object to StreamingQueryProgress JSON and preserve it when QueryProgressEvent JSON and status-store records are serialized and deserialized.
> The object contains the selected trigger type and any relevant configuration:
> {code:json}
> "trigger": {"type": "ProcessingTime", "intervalMs": 1000}
> "trigger": {"type": "RealTime", "batchDurationMs": 300000}
> "trigger": {"type": "AvailableNow"}
> {code}
> The type field is the discriminator for the remaining fields. The configuration names are intentionally type-specific because the values have different meanings:
> * ProcessingTime uses intervalMs to schedule batch starts. A batch's actual execution duration can be shorter or longer than this interval, so naming it batchDurationMs would be inaccurate.
> * Continuous uses intervalMs to schedule checkpoints and does not execute micro-batches.
> * RealTime uses batchDurationMs as each real-time scan's deadline. Consecutive real-time batches start immediately, so this duration also influences batch cadence.
> * AvailableNow and OneTime do not have a duration field.
> Consumers should read type first and then the configuration field defined for that trigger type.
> The field should:
> * support all built-in trigger types;
> * provide stable, queryable type and configuration fields;
> * remain backward-compatible with progress and event records that do not contain it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org