Using Custom log4j with Apex

[Gaurav Gupta <ga...@datatorrent.com>]

When the yarn container runs on Hadoop node, the log4j.properties file is taken from the Hadoop classpath. There could be use cases where user wants to use custom log4j.properties file. 

There are two ways of setting custom log4j.properties in Apex

1. Setting custom log4j.properties at the Application level:  User needs to set following attribute
<property>
  <name>dt.attr.CONTAINER_JVM_OPTIONS</name>
  <value>-Dlog4j.configuration=custom_log4j.properties</value>
</property>

This will make sure that custom log4j properties is used for all containers including Application Master.

2. Setting custom log4j.properties at an individual operator level: User needs to set the following attribute at an operator level

<property>
  <name>dt.operator.<OPERATOR_NAME>.attr.JVM_OPTIONS</name>
  <value> -Dlog4j.configuration=custom_log4j.properties</value>
</property>

This is set the custom log4j properties only on the container that is hosting the operator. 

Make sure that custom_log4j.properties is part of your application jar and is located under src/main/resources

Following are examples of how custom log4j properties file should look like

1. Writing to a file
log4j.rootLogger=${hadoop.root.logger} // this is set to INFO / DEBUG, RFA

log4j.appender.RFA=org.apache.log4j.RollingFileAppender

log4j.appender.RFA.layout=org.apache.log4j.PatternLayout

log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c{2} %M - %m%n

log4j.appender.RFA.File=${hadoop.log.dir}/${hadoop.log.file}

2. Writing to Console

log4j.rootLogger=${hadoop.root.logger} // this is set to INFO / DEBUG, RFA

log4j.appender.RFA=org.apache.log4j.ConsoleAppender

log4j.appender.RFA.layout=org.apache.log4j.PatternLayout

log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c{2} %M - %m%n




Thanks
- Gaurav