[jira] [Created] (SAMZA-350) Allow dynamic log level toggling

["Chris Riccomini (JIRA)" <ji...@apache.org>]

Chris Riccomini created SAMZA-350:
-------------------------------------

             Summary: Allow dynamic log level toggling
                 Key: SAMZA-350
                 URL: https://issues.apache.org/jira/browse/SAMZA-350
             Project: Samza
          Issue Type: Bug
          Components: container
    Affects Versions: 0.7.0
            Reporter: Chris Riccomini


Samza uses SLF4J for logging. We do not expose any mechanism to dynamically change the log level at runtime. I'm not even sure if SLF4J can do this because the log level settings are usually implementation-specific.

Log4J does allow this via the setLevel API. By default in Log4J 2, JMX is already enabled. The hello-samza app shows how to use Log4J 1.2.*, and we use Log4J 1.2.* at LinkedIn.

We should figure out how to implement this. Ideally, we'd do it at the SLF4J level so we don't break the log-implementation-independence that we get with SLF4J. Unfortunately, it doesn't seem that SLF4J supports this. This leaves us with log-implementation-specific support.

I propose in this ticket that we just write Log4J 1.2.* specific implementation that calls:

{code}
Logger.getRootLogger().setLevel(...);
{code}

The second question is how to implement the RPC. There are many ways. We could use the ConfigLog (SAMZA-348), some other control stream, an HTTP endpoint in the containers, or JMX.

My preference is to use JMX for this. It's nice and simple. The ConfigLog approach seems appealing, but it would either require containers to restart (not ideal if you're trying to debug an issue as it's happening) or would require all containers to listen to the ConfigLog topic (seems a big commitment if we want to keep containers light-weight).

The third question is where this code should live. This code should be run in all containers and the YARN AM. I am favor having this in its own module (samza-log4j) just so we don't have to introduce a Log4J dependency to samza-core, which we've thus far been able to avoid. A reasonable location for this seems to be in a TaskLifecycleListener, which could setup the MBean in the beforeInit method. Unfortunately, the YARN AM does not use TaskLifecycleListener. We could write a SamzaAppMasterLifecycleListener interface, and implement the Log4J MBean for both.

There might also be a better place for this to live. An argument could be made that this should be part of Samza-proper, and that we should just switch to Log4J. Another argument could be made that we need a better lifecycle interface, or some generic plugin thingy to let us run this kind of logic.



--
This message was sent by Atlassian JIRA
(v6.2#6252)