You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jevon Cowell (Jira)" <ji...@apache.org> on 2025/04/07 04:56:00 UTC

[jira] [Created] (SPARK-51731) Introduce a new callback "onQueryTriggerStart" to StreamingQueryListener

Jevon Cowell created SPARK-51731:
------------------------------------

             Summary: Introduce a new callback "onQueryTriggerStart" to StreamingQueryListener
                 Key: SPARK-51731
                 URL: https://issues.apache.org/jira/browse/SPARK-51731
             Project: Spark
          Issue Type: Improvement
          Components: Connect, Java API, PySpark, Spark Core, Structured Streaming
    Affects Versions: 4.1.0
            Reporter: Jevon Cowell


Currently, {{{}StreamingQueryListener{}}}s does not notify users when a query is triggered; it only provides updates when a query starts, progresses, becomes idle, or terminates. By introducing a a new event callback that is emitted whenever a query is triggered, users will be informed of this occurrence, allowing for a variety of use cases.

In my use case, I plan to use this new event along with {{QueryProgress}} and {{QueryIdle}} to develop a Maintenance Job Orchestration System. This system will programmatically submit and execute Spark commands, determining the eligibility of maintenance job executions based on the activity status of streaming queries.

I've already started an initial implementation on my own [fork|https://github.com/JevonCowell/spark/pull/1] but before proceeding with writing test to verify functionality, I want to ensure this is something apache folks are interested in having added in the first place. 

Note that for a feature like this I'm: 
 * Ensuring there is backwards compatibility so that any folks with their own custom SparkQueryListener won't break upon upgrading.  
 * Disabling by default to not add unnecessary load to folks who are just upgrading their dependencies 
 * Aiming to set a sane interval to determine how often an event like this should be emitted per query 

I tired my best to see if a request like this had come up before but was unable to. If it has please link me back to that ticket. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org