[jira] [Commented] (SAMZA-503) Lag gauge very slow to update for slow jobs

["Roger Hoover (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265750#comment-14265750 ] 

Roger Hoover commented on SAMZA-503:
------------------------------------

>From [~criccomini]:  In the BrokerProxy, we currently just sleep if we
don't need to fetch messages. We could do a fetch simply to get the latest
offset, update the gauges, then sleep. This will still likely have a 1-2s
latency, but I think this is reasonable.

> Lag gauge very slow to update for slow jobs
> -------------------------------------------
>
>                 Key: SAMZA-503
>                 URL: https://issues.apache.org/jira/browse/SAMZA-503
>             Project: Samza
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.8.0
>         Environment: Mac OS X, Oracle Java 7, ProcessJobFactory
>            Reporter: Roger Hoover
>
> For slow jobs, the KafkaSystemConsumerMetrics.%s-%s-messages-behind-high-watermark) gauge does not get updated very often.
> To reproduce:
> * Create a job that processes one message and sleeps for 5 seconds
> * Create it's input topic but do not populate it yet
> * Start the job
> * Load 1000s of messages to it's input topic.  You can keep adding messages with a "wait -n 1 <kafka console producer command>"
> What happens:
> * Run jconsole to view the JMX metrics
> * The %s-%s-messages-behind-high-watermark gauge will stay at 0 for a LONG time (~10 minutes?) before finally updating.
> What should happen:
> * The gauge should get updated at a reasonable interval (a least every few seconds)
> I think what's happening is that the BrokerProxy only updates the high watermark when a consumer is ready for more messages.  When the job is so slow, this rarely happens to the metric doesn't get updated. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)