[jira] [Comment Edited] (KAFKA-4829) Improve logging of StreamTask commits

["Steven Schlansker (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-4829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050793#comment-16050793 ] 

Steven Schlansker edited comment on KAFKA-4829 at 6/15/17 5:10 PM:
-------------------------------------------------------------------

This ticket is talking about application logs; the link seems to refer to database logs, not at all the same thing.  Or am I missing something?


was (Author: stevenschlansker):
This is talking about application logs; the link seems to refer to database logs, not at all the same thing.  Or am I missing something?

> Improve logging of StreamTask commits
> -------------------------------------
>
>                 Key: KAFKA-4829
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4829
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>    Affects Versions: 0.10.2.0
>            Reporter: Steven Schlansker
>            Priority: Minor
>              Labels: user-experience
>
> Currently I see this every commit interval:
> {code}
> 2017-02-28T21:27:16.659Z INFO <> [StreamThread-1] o.a.k.s.p.internals.StreamThread - stream-thread [StreamThread-1] Committing task StreamTask 1_31
> 2017-02-28T21:27:16.659Z INFO <> [StreamThread-1] o.a.k.s.p.internals.StreamThread - stream-thread [StreamThread-1] Committing task StreamTask 2_31
> {code}
> We have ~10 tasks in our topology, 4 topics, and 32 partitions per topic.
> This means every commit interval we log a few hundred lines of the above
> which is an order of magnitude chattier than anything else in the log
> during normal operations.
> To improve visibility of important messages, we should reduce the chattiness of normal commits and highlight abnormal commits.  An example proposal:
> existing message is fine at TRACE level for diagnostics
> {{TRACE o.a.k.s.p.i.StreamThread - Committing task StreamTask 1_31}}
> normal fast case, wrap them all up into one summary line
> {{INFO o.a.k.s.p.i.StreamThreads - 64 stream tasks committed in 25ms}}
> some kind of threshold / messaging in case it doesn't complete quickly or logs an exception
> {{ERROR o.a.k.s.p.i.StreamThread - StreamTask 1_32 did not commit in 100ms}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)