[jira] [Commented] (KAFKA-7800) Extend Admin API to support dynamic application log levels

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-7800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899144#comment-16899144 ] 

ASF GitHub Bot commented on KAFKA-7800:
---------------------------------------

gwenshap commented on pull request #6903: KAFKA-7800: Dynamic log levels admin API (KIP-412)
URL: https://github.com/apache/kafka/pull/6903
 
 
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Extend Admin API to support dynamic application log levels
> ----------------------------------------------------------
>
>                 Key: KAFKA-7800
>                 URL: https://issues.apache.org/jira/browse/KAFKA-7800
>             Project: Kafka
>          Issue Type: New Feature
>            Reporter: Stanislav Kozlovski
>            Assignee: Stanislav Kozlovski
>            Priority: Major
>
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-412%3A+Extend+Admin+API+to+support+dynamic+application+log+levels]
> Logging is a critical part of any system's infrastructure. It is the most direct way of observing what is happening with a system. In the case of issues, it helps us diagnose the problem quickly which in turn helps lower the [MTTR|http://enterprisedevops.org/article/devops-metric-mean-time-to-recovery-mttr-definition-and-reasoning].
> Kafka supports application logging via the log4j library and outputs messages in various log levels (TRACE, DEBUG, INFO, WARN, ERROR). Log4j is a rich library that supports fine-grained logging configurations (e.g use INFO-level logging in {{kafka.server.ReplicaManager}} and use DEBUG-level in {{kafka.server.KafkaApis}}).
> This is statically configurable through the [log4j.properties|https://github.com/apache/kafka/blob/trunk/config/log4j.properties] file which gets read once at broker start-up.
> A problem with this static configuration is that we cannot alter the log levels when a problem arises. It is severely impractical to edit a properties file and restart all brokers in order to gain visibility of a problem taking place in production.
> It would be very useful if we support dynamically altering the log levels at runtime without needing to restart the Kafka process.
> Log4j itself supports dynamically altering the log levels in a programmatic way and Kafka exposes a [JMX API|https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/utils/Log4jController.scala] that lets you alter them. This allows users to change the log levels via a GUI (jconsole) or a CLI (jmxterm) that uses JMX.
> There is one problem with changing log levels through JMX that we hope to address and that is *Ease of Use*:
>  * Establishing a connection - Connecting to a remote process via JMX requires configuring and exposing multiple JMX ports to the outside world. This is a burden on users, as most production deployments may stand behind layers of firewalls and have policies against opening ports. This makes opening the ports and connections in the middle of an incident even more burdensome
>  * Security - JMX and tools around it support authentication and authorization but it is an additional hassle to set up credentials for another system.
>  * Manual process - Changing the whole cluster's log level requires manually connecting to each broker. In big deployments, this is severely impractical and forces users to build tooling around it.
> h4. Proposition
> Ideally, Kafka would support dynamically changing log levels and address all of the aforementioned concerns out of the box.
> We propose extending the IncrementalAlterConfig/DescribeConfig Admin API with functionality for dynamically altering the broker's log level.
> This approach would also pave the way for even finer-grained logging logic (e.g log DEBUG level only for a certain topic) and would allow us to leverage the existing *AlterConfigPolicy* for custom user-defined validation of log-level changes.
> These log-level changes will be *temporary* and reverted on broker restart - we will not persist them anywhere.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)