[jira] [Created] (SPARK-30692) Mechanism to check that all queries of spark structured-streaming are started in case of multiple sink actions.

["Amit (Jira)" <ji...@apache.org>]

Amit  created SPARK-30692:
-----------------------------

             Summary: Mechanism to check that all queries of spark structured-streaming are started in case of multiple sink actions.
                 Key: SPARK-30692
                 URL: https://issues.apache.org/jira/browse/SPARK-30692
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 2.3.2
            Reporter: Amit 


Get the Spark StructuredStreaming job status (start/stop) having multiple sink actions

We are trying to get the status of StructuredStreaming job, below is the requirement

We wanted to push data to a kafkatopic with offset value set to latest, we are using spark-listeners to get the job status, however we observed that listener is invoked because one of the spark query started but complete spark-job isn't actually started as other queries are still initializing, this results in data-loss because we pushed the data to kafka topic and kafka server set the offset inventory value to the latest, as complete spark job is not started yet but listener gets invoked, once spark job is started it didn't consume data from kafka as offset on kafka server has been already set to latest.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org