You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Sam Davarnia (Jira)" <ji...@apache.org> on 2020/12/17 23:13:00 UTC

[jira] [Created] (SPARK-33833) Allow Spark Structured Streaming report Kafka Lag through Burrow

Sam Davarnia created SPARK-33833:
------------------------------------

             Summary: Allow Spark Structured Streaming report Kafka Lag through Burrow
                 Key: SPARK-33833
                 URL: https://issues.apache.org/jira/browse/SPARK-33833
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 3.0.1
            Reporter: Sam Davarnia


Because structured streaming tracks Kafka offset consumption by itself, 
It is not possible to track total Kafka lag using Burrow similar to DStreams

We have used Stream hooks as mentioned [here|https://medium.com/@ronbarabash/how-to-measure-consumer-lag-in-spark-structured-streaming-6c3645e45a37] 

It would be great if Spark supports this feature out of the box.

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org