You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Sebastian Gäde <s1...@hft-leipzig.de> on 2014/05/18 17:57:31 UTC
where to put log4j.properties for logging out of MR job?
Hi,
I'd like to log events within my MR application using Flume writing to a
central log file. I've followed
http://flume.apache.org/releases/content/1.4.0/FlumeUserGuide.html#log4j-appender
putting the stuff into Hadoop's log4j.properties and copying it to
Hadoop's conf directory on all nodes.
However, my in-application events are only logged to stdout (I can see
them using the YARN webapp) but not to Flume. Although I thought
Hadoop's conf directory is added to classpath, it does not seem to work
for me.
Could you please tell me where I should put the log4j.properties?
Cheers
Seb.
Re: where to put log4j.properties for logging out of MR job?
Posted by Ted Yu <yu...@gmail.com>.
Please see the reply from oers in this thread:
http://stackoverflow.com/questions/9081625/override-log4j-properties-in-hadoop
On May 18, 2014, at 10:38 AM, bo yang <bo...@gmail.com> wrote:
> It might be caused by multiple log4j.properties files in your class path (e.g. in different jar files). For example, I find hadoop-mapreduce-client-jobclient-2.4.0-tests.jar in my class path, and there is log4j.properties inside it. I have to manually delete it to get logging written to files.
>
>
> On Sun, May 18, 2014 at 8:57 AM, Sebastian Gäde <s1...@hft-leipzig.de> wrote:
>> Hi,
>>
>> I'd like to log events within my MR application using Flume writing to a central log file. I've followed http://flume.apache.org/releases/content/1.4.0/FlumeUserGuide.html#log4j-appender putting the stuff into Hadoop's log4j.properties and copying it to Hadoop's conf directory on all nodes.
>>
>> However, my in-application events are only logged to stdout (I can see them using the YARN webapp) but not to Flume. Although I thought Hadoop's conf directory is added to classpath, it does not seem to work for me.
>>
>> Could you please tell me where I should put the log4j.properties?
>>
>> Cheers
>> Seb.
>
Re: where to put log4j.properties for logging out of MR job?
Posted by Ted Yu <yu...@gmail.com>.
Please see the reply from oers in this thread:
http://stackoverflow.com/questions/9081625/override-log4j-properties-in-hadoop
On May 18, 2014, at 10:38 AM, bo yang <bo...@gmail.com> wrote:
> It might be caused by multiple log4j.properties files in your class path (e.g. in different jar files). For example, I find hadoop-mapreduce-client-jobclient-2.4.0-tests.jar in my class path, and there is log4j.properties inside it. I have to manually delete it to get logging written to files.
>
>
> On Sun, May 18, 2014 at 8:57 AM, Sebastian Gäde <s1...@hft-leipzig.de> wrote:
>> Hi,
>>
>> I'd like to log events within my MR application using Flume writing to a central log file. I've followed http://flume.apache.org/releases/content/1.4.0/FlumeUserGuide.html#log4j-appender putting the stuff into Hadoop's log4j.properties and copying it to Hadoop's conf directory on all nodes.
>>
>> However, my in-application events are only logged to stdout (I can see them using the YARN webapp) but not to Flume. Although I thought Hadoop's conf directory is added to classpath, it does not seem to work for me.
>>
>> Could you please tell me where I should put the log4j.properties?
>>
>> Cheers
>> Seb.
>
Re: where to put log4j.properties for logging out of MR job?
Posted by Ted Yu <yu...@gmail.com>.
Please see the reply from oers in this thread:
http://stackoverflow.com/questions/9081625/override-log4j-properties-in-hadoop
On May 18, 2014, at 10:38 AM, bo yang <bo...@gmail.com> wrote:
> It might be caused by multiple log4j.properties files in your class path (e.g. in different jar files). For example, I find hadoop-mapreduce-client-jobclient-2.4.0-tests.jar in my class path, and there is log4j.properties inside it. I have to manually delete it to get logging written to files.
>
>
> On Sun, May 18, 2014 at 8:57 AM, Sebastian Gäde <s1...@hft-leipzig.de> wrote:
>> Hi,
>>
>> I'd like to log events within my MR application using Flume writing to a central log file. I've followed http://flume.apache.org/releases/content/1.4.0/FlumeUserGuide.html#log4j-appender putting the stuff into Hadoop's log4j.properties and copying it to Hadoop's conf directory on all nodes.
>>
>> However, my in-application events are only logged to stdout (I can see them using the YARN webapp) but not to Flume. Although I thought Hadoop's conf directory is added to classpath, it does not seem to work for me.
>>
>> Could you please tell me where I should put the log4j.properties?
>>
>> Cheers
>> Seb.
>
Re: where to put log4j.properties for logging out of MR job?
Posted by Ted Yu <yu...@gmail.com>.
Please see the reply from oers in this thread:
http://stackoverflow.com/questions/9081625/override-log4j-properties-in-hadoop
On May 18, 2014, at 10:38 AM, bo yang <bo...@gmail.com> wrote:
> It might be caused by multiple log4j.properties files in your class path (e.g. in different jar files). For example, I find hadoop-mapreduce-client-jobclient-2.4.0-tests.jar in my class path, and there is log4j.properties inside it. I have to manually delete it to get logging written to files.
>
>
> On Sun, May 18, 2014 at 8:57 AM, Sebastian Gäde <s1...@hft-leipzig.de> wrote:
>> Hi,
>>
>> I'd like to log events within my MR application using Flume writing to a central log file. I've followed http://flume.apache.org/releases/content/1.4.0/FlumeUserGuide.html#log4j-appender putting the stuff into Hadoop's log4j.properties and copying it to Hadoop's conf directory on all nodes.
>>
>> However, my in-application events are only logged to stdout (I can see them using the YARN webapp) but not to Flume. Although I thought Hadoop's conf directory is added to classpath, it does not seem to work for me.
>>
>> Could you please tell me where I should put the log4j.properties?
>>
>> Cheers
>> Seb.
>
Re: where to put log4j.properties for logging out of MR job?
Posted by bo yang <bo...@gmail.com>.
It might be caused by multiple log4j.properties files in your class path
(e.g. in different jar files). For example, I find
hadoop-mapreduce-client-jobclient-2.4.0-tests.jar
in my class path, and there is log4j.properties inside it. I have to
manually delete it to get logging written to files.
On Sun, May 18, 2014 at 8:57 AM, Sebastian Gäde <s1...@hft-leipzig.de>wrote:
> Hi,
>
> I'd like to log events within my MR application using Flume writing to a
> central log file. I've followed http://flume.apache.org/
> releases/content/1.4.0/FlumeUserGuide.html#log4j-appender putting the
> stuff into Hadoop's log4j.properties and copying it to Hadoop's conf
> directory on all nodes.
>
> However, my in-application events are only logged to stdout (I can see
> them using the YARN webapp) but not to Flume. Although I thought Hadoop's
> conf directory is added to classpath, it does not seem to work for me.
>
> Could you please tell me where I should put the log4j.properties?
>
> Cheers
> Seb.
>
Re: where to put log4j.properties for logging out of MR job?
Posted by bo yang <bo...@gmail.com>.
It might be caused by multiple log4j.properties files in your class path
(e.g. in different jar files). For example, I find
hadoop-mapreduce-client-jobclient-2.4.0-tests.jar
in my class path, and there is log4j.properties inside it. I have to
manually delete it to get logging written to files.
On Sun, May 18, 2014 at 8:57 AM, Sebastian Gäde <s1...@hft-leipzig.de>wrote:
> Hi,
>
> I'd like to log events within my MR application using Flume writing to a
> central log file. I've followed http://flume.apache.org/
> releases/content/1.4.0/FlumeUserGuide.html#log4j-appender putting the
> stuff into Hadoop's log4j.properties and copying it to Hadoop's conf
> directory on all nodes.
>
> However, my in-application events are only logged to stdout (I can see
> them using the YARN webapp) but not to Flume. Although I thought Hadoop's
> conf directory is added to classpath, it does not seem to work for me.
>
> Could you please tell me where I should put the log4j.properties?
>
> Cheers
> Seb.
>
Re: where to put log4j.properties for logging out of MR job?
Posted by bo yang <bo...@gmail.com>.
It might be caused by multiple log4j.properties files in your class path
(e.g. in different jar files). For example, I find
hadoop-mapreduce-client-jobclient-2.4.0-tests.jar
in my class path, and there is log4j.properties inside it. I have to
manually delete it to get logging written to files.
On Sun, May 18, 2014 at 8:57 AM, Sebastian Gäde <s1...@hft-leipzig.de>wrote:
> Hi,
>
> I'd like to log events within my MR application using Flume writing to a
> central log file. I've followed http://flume.apache.org/
> releases/content/1.4.0/FlumeUserGuide.html#log4j-appender putting the
> stuff into Hadoop's log4j.properties and copying it to Hadoop's conf
> directory on all nodes.
>
> However, my in-application events are only logged to stdout (I can see
> them using the YARN webapp) but not to Flume. Although I thought Hadoop's
> conf directory is added to classpath, it does not seem to work for me.
>
> Could you please tell me where I should put the log4j.properties?
>
> Cheers
> Seb.
>
Re: where to put log4j.properties for logging out of MR job?
Posted by Sebastian Gäde <s1...@hft-leipzig.de>.
Ok, my fault. It is not about the PatternLayout. But deleting the file Flume is currently using without restarting the agent is not a good idea. ;-)
Cheers
Seb.
Am 19.05.2014 um 20:46 schrieb Sebastian Gäde <s1...@hft-leipzig.de>:
> Hi,
>
> thanks, worked for me, (using http://stackoverflow.com/questions/6072389/how-to-create-a-own-appender-in-log4j):
>
> public class MyFlumeAppender extends AppenderSkeleton {
> MyRpcClientFacade client = new MyRpcClientFacade();
>
> public MyFlumeAppender() {
> super();
> String flumeServer = "localhost";
> Configuration conf = new Configuration();
> conf.addResource("app-config.xml");
> flumeServer = conf.get("flume.server");
> client.init(flumeServer, 41415);
> }
>
> @Override
> public void close() {
> client.cleanUp();
> }
>
> @Override
> public boolean requiresLayout() {
> // TODO Auto-generated method stub
> return false;
> }
>
> @Override
> protected void append(LoggingEvent event) {
> //String logger = event.getLoggerName();
> //String level = event.getLevel().toString();
> //Long ts = event.getTimeStamp();
> //client.sendDataToFlume(ts.toString()+" - "+level+" - "+event.getMessage().toString());
> PatternLayout myLayout = new PatternLayout("%d{ISO8601} %-5p %c - %m");
> client.sendDataToFlume(myLayout.format(event));
> }
>
> }
>
> However, when I try to use the PatternLayout for formatting the output, logging falls back to the container logs. If I build a string on my own (uncommenting the first four lines in append) it's working. I know this might be a bit off topic now, but maybe someone has an idea?
>
> Cheers
> Seb.
>
> Am 19.05.2014 11:17, schrieb Harsh J:
>> Task JVMs in MR run with their own TaskLogAppender-configured
>> task-log4j.properties file. This is not overridable or configurable
>> currently.
>>
>> Your best approach at the moment would be to configure your custom
>> appender in code at the beginning of your program.
>>
>> On Sun, May 18, 2014 at 9:27 PM, Sebastian Gäde <s1...@hft-leipzig.de> wrote:
>>> Hi,
>>>
>>> I'd like to log events within my MR application using Flume writing to a
>>> central log file. I've followed
>>> http://flume.apache.org/releases/content/1.4.0/FlumeUserGuide.html#log4j-appender
>>> putting the stuff into Hadoop's log4j.properties and copying it to Hadoop's
>>> conf directory on all nodes.
>>>
>>> However, my in-application events are only logged to stdout (I can see them
>>> using the YARN webapp) but not to Flume. Although I thought Hadoop's conf
>>> directory is added to classpath, it does not seem to work for me.
>>>
>>> Could you please tell me where I should put the log4j.properties?
>>>
>>> Cheers
>>> Seb.
>>
>>
>>
Re: where to put log4j.properties for logging out of MR job?
Posted by Sebastian Gäde <s1...@hft-leipzig.de>.
Ok, my fault. It is not about the PatternLayout. But deleting the file Flume is currently using without restarting the agent is not a good idea. ;-)
Cheers
Seb.
Am 19.05.2014 um 20:46 schrieb Sebastian Gäde <s1...@hft-leipzig.de>:
> Hi,
>
> thanks, worked for me, (using http://stackoverflow.com/questions/6072389/how-to-create-a-own-appender-in-log4j):
>
> public class MyFlumeAppender extends AppenderSkeleton {
> MyRpcClientFacade client = new MyRpcClientFacade();
>
> public MyFlumeAppender() {
> super();
> String flumeServer = "localhost";
> Configuration conf = new Configuration();
> conf.addResource("app-config.xml");
> flumeServer = conf.get("flume.server");
> client.init(flumeServer, 41415);
> }
>
> @Override
> public void close() {
> client.cleanUp();
> }
>
> @Override
> public boolean requiresLayout() {
> // TODO Auto-generated method stub
> return false;
> }
>
> @Override
> protected void append(LoggingEvent event) {
> //String logger = event.getLoggerName();
> //String level = event.getLevel().toString();
> //Long ts = event.getTimeStamp();
> //client.sendDataToFlume(ts.toString()+" - "+level+" - "+event.getMessage().toString());
> PatternLayout myLayout = new PatternLayout("%d{ISO8601} %-5p %c - %m");
> client.sendDataToFlume(myLayout.format(event));
> }
>
> }
>
> However, when I try to use the PatternLayout for formatting the output, logging falls back to the container logs. If I build a string on my own (uncommenting the first four lines in append) it's working. I know this might be a bit off topic now, but maybe someone has an idea?
>
> Cheers
> Seb.
>
> Am 19.05.2014 11:17, schrieb Harsh J:
>> Task JVMs in MR run with their own TaskLogAppender-configured
>> task-log4j.properties file. This is not overridable or configurable
>> currently.
>>
>> Your best approach at the moment would be to configure your custom
>> appender in code at the beginning of your program.
>>
>> On Sun, May 18, 2014 at 9:27 PM, Sebastian Gäde <s1...@hft-leipzig.de> wrote:
>>> Hi,
>>>
>>> I'd like to log events within my MR application using Flume writing to a
>>> central log file. I've followed
>>> http://flume.apache.org/releases/content/1.4.0/FlumeUserGuide.html#log4j-appender
>>> putting the stuff into Hadoop's log4j.properties and copying it to Hadoop's
>>> conf directory on all nodes.
>>>
>>> However, my in-application events are only logged to stdout (I can see them
>>> using the YARN webapp) but not to Flume. Although I thought Hadoop's conf
>>> directory is added to classpath, it does not seem to work for me.
>>>
>>> Could you please tell me where I should put the log4j.properties?
>>>
>>> Cheers
>>> Seb.
>>
>>
>>
Re: where to put log4j.properties for logging out of MR job?
Posted by Sebastian Gäde <s1...@hft-leipzig.de>.
Ok, my fault. It is not about the PatternLayout. But deleting the file Flume is currently using without restarting the agent is not a good idea. ;-)
Cheers
Seb.
Am 19.05.2014 um 20:46 schrieb Sebastian Gäde <s1...@hft-leipzig.de>:
> Hi,
>
> thanks, worked for me, (using http://stackoverflow.com/questions/6072389/how-to-create-a-own-appender-in-log4j):
>
> public class MyFlumeAppender extends AppenderSkeleton {
> MyRpcClientFacade client = new MyRpcClientFacade();
>
> public MyFlumeAppender() {
> super();
> String flumeServer = "localhost";
> Configuration conf = new Configuration();
> conf.addResource("app-config.xml");
> flumeServer = conf.get("flume.server");
> client.init(flumeServer, 41415);
> }
>
> @Override
> public void close() {
> client.cleanUp();
> }
>
> @Override
> public boolean requiresLayout() {
> // TODO Auto-generated method stub
> return false;
> }
>
> @Override
> protected void append(LoggingEvent event) {
> //String logger = event.getLoggerName();
> //String level = event.getLevel().toString();
> //Long ts = event.getTimeStamp();
> //client.sendDataToFlume(ts.toString()+" - "+level+" - "+event.getMessage().toString());
> PatternLayout myLayout = new PatternLayout("%d{ISO8601} %-5p %c - %m");
> client.sendDataToFlume(myLayout.format(event));
> }
>
> }
>
> However, when I try to use the PatternLayout for formatting the output, logging falls back to the container logs. If I build a string on my own (uncommenting the first four lines in append) it's working. I know this might be a bit off topic now, but maybe someone has an idea?
>
> Cheers
> Seb.
>
> Am 19.05.2014 11:17, schrieb Harsh J:
>> Task JVMs in MR run with their own TaskLogAppender-configured
>> task-log4j.properties file. This is not overridable or configurable
>> currently.
>>
>> Your best approach at the moment would be to configure your custom
>> appender in code at the beginning of your program.
>>
>> On Sun, May 18, 2014 at 9:27 PM, Sebastian Gäde <s1...@hft-leipzig.de> wrote:
>>> Hi,
>>>
>>> I'd like to log events within my MR application using Flume writing to a
>>> central log file. I've followed
>>> http://flume.apache.org/releases/content/1.4.0/FlumeUserGuide.html#log4j-appender
>>> putting the stuff into Hadoop's log4j.properties and copying it to Hadoop's
>>> conf directory on all nodes.
>>>
>>> However, my in-application events are only logged to stdout (I can see them
>>> using the YARN webapp) but not to Flume. Although I thought Hadoop's conf
>>> directory is added to classpath, it does not seem to work for me.
>>>
>>> Could you please tell me where I should put the log4j.properties?
>>>
>>> Cheers
>>> Seb.
>>
>>
>>
Re: where to put log4j.properties for logging out of MR job?
Posted by Sebastian Gäde <s1...@hft-leipzig.de>.
Ok, my fault. It is not about the PatternLayout. But deleting the file Flume is currently using without restarting the agent is not a good idea. ;-)
Cheers
Seb.
Am 19.05.2014 um 20:46 schrieb Sebastian Gäde <s1...@hft-leipzig.de>:
> Hi,
>
> thanks, worked for me, (using http://stackoverflow.com/questions/6072389/how-to-create-a-own-appender-in-log4j):
>
> public class MyFlumeAppender extends AppenderSkeleton {
> MyRpcClientFacade client = new MyRpcClientFacade();
>
> public MyFlumeAppender() {
> super();
> String flumeServer = "localhost";
> Configuration conf = new Configuration();
> conf.addResource("app-config.xml");
> flumeServer = conf.get("flume.server");
> client.init(flumeServer, 41415);
> }
>
> @Override
> public void close() {
> client.cleanUp();
> }
>
> @Override
> public boolean requiresLayout() {
> // TODO Auto-generated method stub
> return false;
> }
>
> @Override
> protected void append(LoggingEvent event) {
> //String logger = event.getLoggerName();
> //String level = event.getLevel().toString();
> //Long ts = event.getTimeStamp();
> //client.sendDataToFlume(ts.toString()+" - "+level+" - "+event.getMessage().toString());
> PatternLayout myLayout = new PatternLayout("%d{ISO8601} %-5p %c - %m");
> client.sendDataToFlume(myLayout.format(event));
> }
>
> }
>
> However, when I try to use the PatternLayout for formatting the output, logging falls back to the container logs. If I build a string on my own (uncommenting the first four lines in append) it's working. I know this might be a bit off topic now, but maybe someone has an idea?
>
> Cheers
> Seb.
>
> Am 19.05.2014 11:17, schrieb Harsh J:
>> Task JVMs in MR run with their own TaskLogAppender-configured
>> task-log4j.properties file. This is not overridable or configurable
>> currently.
>>
>> Your best approach at the moment would be to configure your custom
>> appender in code at the beginning of your program.
>>
>> On Sun, May 18, 2014 at 9:27 PM, Sebastian Gäde <s1...@hft-leipzig.de> wrote:
>>> Hi,
>>>
>>> I'd like to log events within my MR application using Flume writing to a
>>> central log file. I've followed
>>> http://flume.apache.org/releases/content/1.4.0/FlumeUserGuide.html#log4j-appender
>>> putting the stuff into Hadoop's log4j.properties and copying it to Hadoop's
>>> conf directory on all nodes.
>>>
>>> However, my in-application events are only logged to stdout (I can see them
>>> using the YARN webapp) but not to Flume. Although I thought Hadoop's conf
>>> directory is added to classpath, it does not seem to work for me.
>>>
>>> Could you please tell me where I should put the log4j.properties?
>>>
>>> Cheers
>>> Seb.
>>
>>
>>
Re: where to put log4j.properties for logging out of MR job?
Posted by Sebastian Gäde <s1...@hft-leipzig.de>.
Hi,
thanks, worked for me, (using
http://stackoverflow.com/questions/6072389/how-to-create-a-own-appender-in-log4j):
public class MyFlumeAppender extends AppenderSkeleton {
MyRpcClientFacade client = new MyRpcClientFacade();
public MyFlumeAppender() {
super();
String flumeServer = "localhost";
Configuration conf = new Configuration();
conf.addResource("app-config.xml");
flumeServer = conf.get("flume.server");
client.init(flumeServer, 41415);
}
@Override
public void close() {
client.cleanUp();
}
@Override
public boolean requiresLayout() {
// TODO Auto-generated method stub
return false;
}
@Override
protected void append(LoggingEvent event) {
//String logger = event.getLoggerName();
//String level = event.getLevel().toString();
//Long ts = event.getTimeStamp();
//client.sendDataToFlume(ts.toString()+" - "+level+" -
"+event.getMessage().toString());
PatternLayout myLayout = new PatternLayout("%d{ISO8601} %-5p %c - %m");
client.sendDataToFlume(myLayout.format(event));
}
}
However, when I try to use the PatternLayout for formatting the output,
logging falls back to the container logs. If I build a string on my own
(uncommenting the first four lines in append) it's working. I know this
might be a bit off topic now, but maybe someone has an idea?
Cheers
Seb.
Am 19.05.2014 11:17, schrieb Harsh J:
> Task JVMs in MR run with their own TaskLogAppender-configured
> task-log4j.properties file. This is not overridable or configurable
> currently.
>
> Your best approach at the moment would be to configure your custom
> appender in code at the beginning of your program.
>
> On Sun, May 18, 2014 at 9:27 PM, Sebastian Gäde <s1...@hft-leipzig.de> wrote:
>> Hi,
>>
>> I'd like to log events within my MR application using Flume writing to a
>> central log file. I've followed
>> http://flume.apache.org/releases/content/1.4.0/FlumeUserGuide.html#log4j-appender
>> putting the stuff into Hadoop's log4j.properties and copying it to Hadoop's
>> conf directory on all nodes.
>>
>> However, my in-application events are only logged to stdout (I can see them
>> using the YARN webapp) but not to Flume. Although I thought Hadoop's conf
>> directory is added to classpath, it does not seem to work for me.
>>
>> Could you please tell me where I should put the log4j.properties?
>>
>> Cheers
>> Seb.
>
>
>
Re: where to put log4j.properties for logging out of MR job?
Posted by Sebastian Gäde <s1...@hft-leipzig.de>.
Hi,
thanks, worked for me, (using
http://stackoverflow.com/questions/6072389/how-to-create-a-own-appender-in-log4j):
public class MyFlumeAppender extends AppenderSkeleton {
MyRpcClientFacade client = new MyRpcClientFacade();
public MyFlumeAppender() {
super();
String flumeServer = "localhost";
Configuration conf = new Configuration();
conf.addResource("app-config.xml");
flumeServer = conf.get("flume.server");
client.init(flumeServer, 41415);
}
@Override
public void close() {
client.cleanUp();
}
@Override
public boolean requiresLayout() {
// TODO Auto-generated method stub
return false;
}
@Override
protected void append(LoggingEvent event) {
//String logger = event.getLoggerName();
//String level = event.getLevel().toString();
//Long ts = event.getTimeStamp();
//client.sendDataToFlume(ts.toString()+" - "+level+" -
"+event.getMessage().toString());
PatternLayout myLayout = new PatternLayout("%d{ISO8601} %-5p %c - %m");
client.sendDataToFlume(myLayout.format(event));
}
}
However, when I try to use the PatternLayout for formatting the output,
logging falls back to the container logs. If I build a string on my own
(uncommenting the first four lines in append) it's working. I know this
might be a bit off topic now, but maybe someone has an idea?
Cheers
Seb.
Am 19.05.2014 11:17, schrieb Harsh J:
> Task JVMs in MR run with their own TaskLogAppender-configured
> task-log4j.properties file. This is not overridable or configurable
> currently.
>
> Your best approach at the moment would be to configure your custom
> appender in code at the beginning of your program.
>
> On Sun, May 18, 2014 at 9:27 PM, Sebastian Gäde <s1...@hft-leipzig.de> wrote:
>> Hi,
>>
>> I'd like to log events within my MR application using Flume writing to a
>> central log file. I've followed
>> http://flume.apache.org/releases/content/1.4.0/FlumeUserGuide.html#log4j-appender
>> putting the stuff into Hadoop's log4j.properties and copying it to Hadoop's
>> conf directory on all nodes.
>>
>> However, my in-application events are only logged to stdout (I can see them
>> using the YARN webapp) but not to Flume. Although I thought Hadoop's conf
>> directory is added to classpath, it does not seem to work for me.
>>
>> Could you please tell me where I should put the log4j.properties?
>>
>> Cheers
>> Seb.
>
>
>
Re: where to put log4j.properties for logging out of MR job?
Posted by Sebastian Gäde <s1...@hft-leipzig.de>.
Hi,
thanks, worked for me, (using
http://stackoverflow.com/questions/6072389/how-to-create-a-own-appender-in-log4j):
public class MyFlumeAppender extends AppenderSkeleton {
MyRpcClientFacade client = new MyRpcClientFacade();
public MyFlumeAppender() {
super();
String flumeServer = "localhost";
Configuration conf = new Configuration();
conf.addResource("app-config.xml");
flumeServer = conf.get("flume.server");
client.init(flumeServer, 41415);
}
@Override
public void close() {
client.cleanUp();
}
@Override
public boolean requiresLayout() {
// TODO Auto-generated method stub
return false;
}
@Override
protected void append(LoggingEvent event) {
//String logger = event.getLoggerName();
//String level = event.getLevel().toString();
//Long ts = event.getTimeStamp();
//client.sendDataToFlume(ts.toString()+" - "+level+" -
"+event.getMessage().toString());
PatternLayout myLayout = new PatternLayout("%d{ISO8601} %-5p %c - %m");
client.sendDataToFlume(myLayout.format(event));
}
}
However, when I try to use the PatternLayout for formatting the output,
logging falls back to the container logs. If I build a string on my own
(uncommenting the first four lines in append) it's working. I know this
might be a bit off topic now, but maybe someone has an idea?
Cheers
Seb.
Am 19.05.2014 11:17, schrieb Harsh J:
> Task JVMs in MR run with their own TaskLogAppender-configured
> task-log4j.properties file. This is not overridable or configurable
> currently.
>
> Your best approach at the moment would be to configure your custom
> appender in code at the beginning of your program.
>
> On Sun, May 18, 2014 at 9:27 PM, Sebastian Gäde <s1...@hft-leipzig.de> wrote:
>> Hi,
>>
>> I'd like to log events within my MR application using Flume writing to a
>> central log file. I've followed
>> http://flume.apache.org/releases/content/1.4.0/FlumeUserGuide.html#log4j-appender
>> putting the stuff into Hadoop's log4j.properties and copying it to Hadoop's
>> conf directory on all nodes.
>>
>> However, my in-application events are only logged to stdout (I can see them
>> using the YARN webapp) but not to Flume. Although I thought Hadoop's conf
>> directory is added to classpath, it does not seem to work for me.
>>
>> Could you please tell me where I should put the log4j.properties?
>>
>> Cheers
>> Seb.
>
>
>
Re: where to put log4j.properties for logging out of MR job?
Posted by Sebastian Gäde <s1...@hft-leipzig.de>.
Hi,
thanks, worked for me, (using
http://stackoverflow.com/questions/6072389/how-to-create-a-own-appender-in-log4j):
public class MyFlumeAppender extends AppenderSkeleton {
MyRpcClientFacade client = new MyRpcClientFacade();
public MyFlumeAppender() {
super();
String flumeServer = "localhost";
Configuration conf = new Configuration();
conf.addResource("app-config.xml");
flumeServer = conf.get("flume.server");
client.init(flumeServer, 41415);
}
@Override
public void close() {
client.cleanUp();
}
@Override
public boolean requiresLayout() {
// TODO Auto-generated method stub
return false;
}
@Override
protected void append(LoggingEvent event) {
//String logger = event.getLoggerName();
//String level = event.getLevel().toString();
//Long ts = event.getTimeStamp();
//client.sendDataToFlume(ts.toString()+" - "+level+" -
"+event.getMessage().toString());
PatternLayout myLayout = new PatternLayout("%d{ISO8601} %-5p %c - %m");
client.sendDataToFlume(myLayout.format(event));
}
}
However, when I try to use the PatternLayout for formatting the output,
logging falls back to the container logs. If I build a string on my own
(uncommenting the first four lines in append) it's working. I know this
might be a bit off topic now, but maybe someone has an idea?
Cheers
Seb.
Am 19.05.2014 11:17, schrieb Harsh J:
> Task JVMs in MR run with their own TaskLogAppender-configured
> task-log4j.properties file. This is not overridable or configurable
> currently.
>
> Your best approach at the moment would be to configure your custom
> appender in code at the beginning of your program.
>
> On Sun, May 18, 2014 at 9:27 PM, Sebastian Gäde <s1...@hft-leipzig.de> wrote:
>> Hi,
>>
>> I'd like to log events within my MR application using Flume writing to a
>> central log file. I've followed
>> http://flume.apache.org/releases/content/1.4.0/FlumeUserGuide.html#log4j-appender
>> putting the stuff into Hadoop's log4j.properties and copying it to Hadoop's
>> conf directory on all nodes.
>>
>> However, my in-application events are only logged to stdout (I can see them
>> using the YARN webapp) but not to Flume. Although I thought Hadoop's conf
>> directory is added to classpath, it does not seem to work for me.
>>
>> Could you please tell me where I should put the log4j.properties?
>>
>> Cheers
>> Seb.
>
>
>
Re: where to put log4j.properties for logging out of MR job?
Posted by Harsh J <ha...@cloudera.com>.
Task JVMs in MR run with their own TaskLogAppender-configured
task-log4j.properties file. This is not overridable or configurable
currently.
Your best approach at the moment would be to configure your custom
appender in code at the beginning of your program.
On Sun, May 18, 2014 at 9:27 PM, Sebastian Gäde <s1...@hft-leipzig.de> wrote:
> Hi,
>
> I'd like to log events within my MR application using Flume writing to a
> central log file. I've followed
> http://flume.apache.org/releases/content/1.4.0/FlumeUserGuide.html#log4j-appender
> putting the stuff into Hadoop's log4j.properties and copying it to Hadoop's
> conf directory on all nodes.
>
> However, my in-application events are only logged to stdout (I can see them
> using the YARN webapp) but not to Flume. Although I thought Hadoop's conf
> directory is added to classpath, it does not seem to work for me.
>
> Could you please tell me where I should put the log4j.properties?
>
> Cheers
> Seb.
--
Harsh J
Re: where to put log4j.properties for logging out of MR job?
Posted by Harsh J <ha...@cloudera.com>.
Task JVMs in MR run with their own TaskLogAppender-configured
task-log4j.properties file. This is not overridable or configurable
currently.
Your best approach at the moment would be to configure your custom
appender in code at the beginning of your program.
On Sun, May 18, 2014 at 9:27 PM, Sebastian Gäde <s1...@hft-leipzig.de> wrote:
> Hi,
>
> I'd like to log events within my MR application using Flume writing to a
> central log file. I've followed
> http://flume.apache.org/releases/content/1.4.0/FlumeUserGuide.html#log4j-appender
> putting the stuff into Hadoop's log4j.properties and copying it to Hadoop's
> conf directory on all nodes.
>
> However, my in-application events are only logged to stdout (I can see them
> using the YARN webapp) but not to Flume. Although I thought Hadoop's conf
> directory is added to classpath, it does not seem to work for me.
>
> Could you please tell me where I should put the log4j.properties?
>
> Cheers
> Seb.
--
Harsh J
Re: where to put log4j.properties for logging out of MR job?
Posted by bo yang <bo...@gmail.com>.
It might be caused by multiple log4j.properties files in your class path
(e.g. in different jar files). For example, I find
hadoop-mapreduce-client-jobclient-2.4.0-tests.jar
in my class path, and there is log4j.properties inside it. I have to
manually delete it to get logging written to files.
On Sun, May 18, 2014 at 8:57 AM, Sebastian Gäde <s1...@hft-leipzig.de>wrote:
> Hi,
>
> I'd like to log events within my MR application using Flume writing to a
> central log file. I've followed http://flume.apache.org/
> releases/content/1.4.0/FlumeUserGuide.html#log4j-appender putting the
> stuff into Hadoop's log4j.properties and copying it to Hadoop's conf
> directory on all nodes.
>
> However, my in-application events are only logged to stdout (I can see
> them using the YARN webapp) but not to Flume. Although I thought Hadoop's
> conf directory is added to classpath, it does not seem to work for me.
>
> Could you please tell me where I should put the log4j.properties?
>
> Cheers
> Seb.
>
Re: where to put log4j.properties for logging out of MR job?
Posted by Harsh J <ha...@cloudera.com>.
Task JVMs in MR run with their own TaskLogAppender-configured
task-log4j.properties file. This is not overridable or configurable
currently.
Your best approach at the moment would be to configure your custom
appender in code at the beginning of your program.
On Sun, May 18, 2014 at 9:27 PM, Sebastian Gäde <s1...@hft-leipzig.de> wrote:
> Hi,
>
> I'd like to log events within my MR application using Flume writing to a
> central log file. I've followed
> http://flume.apache.org/releases/content/1.4.0/FlumeUserGuide.html#log4j-appender
> putting the stuff into Hadoop's log4j.properties and copying it to Hadoop's
> conf directory on all nodes.
>
> However, my in-application events are only logged to stdout (I can see them
> using the YARN webapp) but not to Flume. Although I thought Hadoop's conf
> directory is added to classpath, it does not seem to work for me.
>
> Could you please tell me where I should put the log4j.properties?
>
> Cheers
> Seb.
--
Harsh J
Re: where to put log4j.properties for logging out of MR job?
Posted by Harsh J <ha...@cloudera.com>.
Task JVMs in MR run with their own TaskLogAppender-configured
task-log4j.properties file. This is not overridable or configurable
currently.
Your best approach at the moment would be to configure your custom
appender in code at the beginning of your program.
On Sun, May 18, 2014 at 9:27 PM, Sebastian Gäde <s1...@hft-leipzig.de> wrote:
> Hi,
>
> I'd like to log events within my MR application using Flume writing to a
> central log file. I've followed
> http://flume.apache.org/releases/content/1.4.0/FlumeUserGuide.html#log4j-appender
> putting the stuff into Hadoop's log4j.properties and copying it to Hadoop's
> conf directory on all nodes.
>
> However, my in-application events are only logged to stdout (I can see them
> using the YARN webapp) but not to Flume. Although I thought Hadoop's conf
> directory is added to classpath, it does not seem to work for me.
>
> Could you please tell me where I should put the log4j.properties?
>
> Cheers
> Seb.
--
Harsh J