[jira] [Commented] (TEZ-2222) Investigate moving to log4j2 for logging

["Yingda Chen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-2222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680589#comment-16680589 ] 

Yingda Chen commented on TEZ-2222:
----------------------------------

[~jeagles] I would like to provide some more context about our usage and development on Tez. Currently we have forked a version of Tez and have removed its dependency on Hadoop entirely since we are using it to run on top of our own proprietary resource manager other than YARN. In other words, we are using Tez as a standalone AM without interacting with YARN.

In this context, it would be a bit hard for us to effectively and thoroughly test how upgrading to log4j2 will impact its interactions (dependency tree etc.) with YARN. In addition, since a large portion of this JIRA would be coordination work, I think it would be much more effective if someone from Hortonworks can coordinate this upgrade instead.

That said, we indeed have upgraded the logging framework to log4j2 in our version of Tez, and have seen very noticeable improvement, in terms of both performance and GC – as promised by log4j2. Such improvements are especially significant in large-scale job,  even more so in large-scale job with a lot of relatively-short-live tasks.

Finally, we are in the middle of contributing some of our work on Tez back to the community, beginning with TEZ-3997. Other features we are hoping to work on also include support of more dynamic DAG (conditional graph), improvement on node tracking and management, and hopefully a resource/runtime abstraction layer that would allow Tez-as-a-library to run on different distributed framework, including YARN/Mesos/K8S etc, we will follow up with separate JIRAs.

> Investigate moving to log4j2 for logging
> ----------------------------------------
>
>                 Key: TEZ-2222
>                 URL: https://issues.apache.org/jira/browse/TEZ-2222
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Siddharth Seth
>            Priority: Blocker
>
> Via slf4j.
> Some bits to keep in mind
> - We have explicit code which rotates logs using direct log4j12 APIs. This should keep working. I believe the log4j2 APIs are different here
> - API compatibility between log4j12 / log4j2 can be problematic - if both end up on the classpath (I believe the APIs are different)
> - Hadoop dist includes a slf4j-log4j12 binding. Changing the default can result in sl4j-log4j12 and slf4j-log4j2 to co-exist by default - which could be problematic. Needs investigation.
> End of the day, we will likely need an option to use either of the two.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)