Read duration, Write duration, Processing Time metrics

[Melika Ghiasi <me...@gmail.com>]

Hello

I have several *Structured Streaming Spark applications*, and I want to
measure three key performance metrics *per micro-batch*:

   1.

   *Write duration* – the total time taken to write data to an output sink
   (e.g., a Delta Lake sink).
   2.

   *Processing time* – the time spent performing operations like
   aggregations or joins within each micro-batch.
   3.

   *Read duration* – the time spent reading from a streaming source (e.g.,
   Kafka) while data is arriving.

How can I access these metrics?

I have experimented with writing a custom listener using both the
SparkListener and the StreamingQueryListener classes. The most relevant
metrics I’ve found are:

   -

   *For processing time*: CPU time and runtime metrics available in
   stageInfo.taskMetrics, which are aggregated across all tasks per stage.
   -

   *For write duration*: ShuffleWriteMetrics.writeTime, which unfortunately
   only reflects time spent writing shuffle data, *not* writing to external
   sinks.
   -

   *For read duration*: QueryProgressEvent.durationMs.latestOffset, which
   applies only to certain streaming sources like Kafka, and doesn't reflect
   the actual read duration across all sources.

As a result, I haven’t found a reliable or complete set of metrics that
accurately report *write, read, and processing durations* separately for
each micro-batch.

Can anyone provide guidance or solutions for how to extract or approximate
these metrics?