You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@gobblin.apache.org by "Sudarshan Vasudevan (Jira)" <ji...@apache.org> on 2020/03/17 05:24:00 UTC

[jira] [Created] (GOBBLIN-1087) Track and report histogram of observed lag from Gobblin Kafka pipeline

Sudarshan Vasudevan created GOBBLIN-1087:
--------------------------------------------

             Summary: Track and report histogram of observed lag from Gobblin Kafka pipeline
                 Key: GOBBLIN-1087
                 URL: https://issues.apache.org/jira/browse/GOBBLIN-1087
             Project: Apache Gobblin
          Issue Type: Improvement
          Components: gobblin-kafka
    Affects Versions: 0.15.0
            Reporter: Sudarshan Vasudevan
            Assignee: Shirshanka Das
             Fix For: 0.15.0


In this PR, we instrument the KafkaExtractor to track the observed latency of Kafka consumer records processed by the pipeline. Here, observed latency is measured as the time difference between processing time of the record and the original creation time. The latency distribution is tracked in an HdrHistogram, which is serialized into a string when emitted as part of a GobblinTrackingEvent.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)