[jira] [Resolved] (GOBBLIN-452) Logging related Improvement in Gobblin Cluster

["Abhishek Tiwari (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/GOBBLIN-452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Abhishek Tiwari resolved GOBBLIN-452.
-------------------------------------
       Resolution: Fixed
    Fix Version/s: 0.13.0

Issue resolved by pull request #2362
[https://github.com/apache/incubator-gobblin/pull/2362]

> Logging related Improvement in Gobblin Cluster
> ----------------------------------------------
>
>                 Key: GOBBLIN-452
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-452
>             Project: Apache Gobblin
>          Issue Type: Improvement
>            Reporter: Vicky Kak
>            Assignee: Vicky Kak
>            Priority: Trivial
>             Fix For: 0.13.0
>
>
> Following are the logging related improvement that need to be done in Gobblin Cluster Implementation
> 1) I have running the GAAS and Standalone cluster on the same machine and from the same distribution, this will be typically needed for quick setup. Since I have been starting the GAAS and Standalone master on same distribution, they both are directing the logs to the same master.out file leading to overlap of the logging details from the GAAS and standalone master. I have changed the logging file from master.out to clustermaster.out on my local set up by changing the $GOBBLIN_HOME/bin/gobblin-cluster-master.sh as
> nohup $COMMAND >clustermaster.out 2>&1 & echo $! > $PID
> We better make the changes in the distribution.
> 2) The log4j logging configuration is dynamically controlled in the standalone/worker implementation, it does not work by default.I looked at how the log4j configurations are being controlled in other modes, it is done via the bootstrap scripts e.g gobblin-aws.sh as LOG4J_PATH=file://${FWDIR_CONF}/log4j-aws.properties COMMAND="$JAVA_HOME/bin/java -cp $CLASSPATH $JVM_FLAGS gobblin.aws.GobblinAWSClusterLauncher -D log4j.configuration=$LOG4J_PATH" I see the log4j configurations similarly being configured in gobblin-standalone.sh too COMMAND+="-Dlog4j.configuration=file://$FWDIR_CONF/log4j-standalone.xml " I did made the similar changes for the gobblin-service.sh as LOG4J_PATH=file://${FWDIR_CONF}/log4j-cluster.properties COMMAND="$JAVA_HOME/bin/java -Dlog4j.debug -Dlog4j.configuration=$LOG4J_PATH -cp $CLASSPATH $JVM_FLAGS gobblin.service.modules.core.GobblinServiceManager --service_name $SERVICE_NAME $LOG_ARGS" This was done because the log4j configuration for the GAAS which should have been taken from $GOBBLIN_HOME/conf/service/log4j-cluster.properties was not being taken from there, it was taken from the $GOBBLIN_HOME/lib/generator-2.6.0.jar. We should keep the consistent model of loading the log4j, for the standalone cluster the log4j configurations are being loaded via code and for the other gobblin components(modes) it is via the configuration in the bootstrap scripts. We should have it consistent and I think having it in the bootstrap scripts via -Dlog4j.configuration is good option. I have to copy the log4j-cluster.properties into the GOBBLIN_HOME/bin for running the Standalone cluster master/worker node. We need to fix these log4j configrations issues.
>  
> This is already discussed here 
> [https://mail-archives.apache.org/mod_mbox/incubator-gobblin-user/201801.mbox/%3CCAPaCpY-QT5e%3DJm-94k2aZOLWq5crOLbPYK4KusAmEgUvQYHa0w%40mail.gmail.com%3E]
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)