You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by ja...@marketingscience.co on 2015/06/26 19:39:27 UTC
Best way to log from inside a Samza task?
Hello,
I am working on a basic Samza task that pulls from one Kafka topic and writes to another. This task runs in Yarn but the Output topic does not contain any data.
In order to troubleshoot this more effectively I would like to log the incoming message as my example below. Ideally, I would like to be able to see the log messages in Yarn, maybe in the .out files in the /logs directory.
Any advice is appreciated.
Here is my task:
package com.project.samza.tasks;
import java.util.Map;
import org.apache.samza.system.IncomingMessageEnvelope;
import org.apache.samza.system.OutgoingMessageEnvelope;
import org.apache.samza.system.SystemStream;
import org.apache.samza.task.MessageCollector;
import org.apache.samza.task.StreamTask;
import org.apache.samza.task.TaskCoordinator;
public class exampleStreamTask implements StreamTask {
private static final SystemStream OUTPUT_STREAM = new SystemStream(“kafka”, “new-topic-test”);
@Override
public void process(IncomingMessageEnvelope envelope,
MessageCollector collector,
TaskCoordinator coordinator) {
String msg = (String) envelope.getMessage();
System.out.println(msg);
collector.send(new OutgoingMessageEnvelope(OUTPUT_STREAM, msg));
}
}
Thanks,
Jason
Re: Best way to log from inside a Samza task?
Posted by Rick Mangi <ri...@chartbeat.com>.
Hi Jason,
It depends on how you’re deploying. I deploy on yarn via cloudera and my logging statements wind up in the container log for each node.
You need to instantiate a logger for your class before writing to it. Look at the log4j docs to see the right way to do that.
Rick
> On Jul 1, 2015, at 2:58 PM, jason@marketingscience.co wrote:
>
> Rick,
>
> From your example, what is the correct way to call the logger in the task? Something like:
>
> import java.util.Map;
> import org.apache.samza.system.IncomingMessageEnvelope;
> import org.apache.samza.system.OutgoingMessageEnvelope;
> import org.apache.samza.system.SystemStream;
> import org.apache.samza.task.MessageCollector;
> import org.apache.samza.task.StreamTask;
> import org.apache.samza.task.TaskCoordinator;
>
> public class uimodPageloadStreamTask implements StreamTask {
> private static final SystemStream OUTPUT_STREAM = new SystemStream(“kafka”, “kafka-output-test”);
>
> @Override
> public void process(IncomingMessageEnvelope envelope,
> MessageCollector collector,
> TaskCoordinator coordinator) {
> logger.info(envelope.getMessage());
> collector.send(new OutgoingMessageEnvelope(OUTPUT_STREAM, text));
> }
> }
>
> Also, in what log file would the log messages end up? /logs/stdout for that application container?
>
> I got System.out.println() to work similarly but it seems to be doing some formatting of the messages.
>
> I am trying to consume messages from one Kafka topic and produce to another. While I can use zookeeper to see the messages in the originating topic they never make it to the destination and I am trying to find out why.
>
> Thanks for your help,
>
> Jason
>
>
> On Friday, Jun 26, 2558 at 12:03, Rick Mangi <rick@chartbeat.com <ma...@chartbeat.com>>, wrote:
> If you do something like this in your log4j.xml
>
> <root>
> <priority value="INFO" />
> <appender-ref ref="RollingAppender" />
> </root>
>
> <logger name="cbsamza" additivity="false">
> <level value=“DEBUG" />
> <appender-ref ref="RollingAppender" />
> </logger>
>
> the root controls samza’s logging and the logger controls your own… I haven’t managed to get the imx configuration working yet.
>
>
>
>
> > On Jun 26, 2015, at 1:50 PM, jason@marketingscience.co wrote:
> >
> > I was almost there. Got it now. Thanks for your help Rick.
> >
> >
> >
> >
> > Cheers,
> >
> >
> >
> >
> > Jason
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Friday, Jun 26, 2558 at 11:43, Rick Mangi <ri...@chartbeat.com>, wrote:
> > Hey Jason,
> >
> >
> > If you configure log4j as described here: http://samza.apache.org/learn/documentation/0.9/jobs/logging.html <http://samza.apache.org/learn/documentation/0.9/jobs/logging.html>
> >
> >
> > Your log statements will wind up in the samza-container logs which you can get to via the application master gui.
> >
> >
> > hth,
> >
> >
> > Rick
> >
> >
> >
> >> On Jun 26, 2015, at 1:39 PM, jason@marketingscience.co wrote:
> >
> >>
> >
> >> Hello,
> >
> >>
> >
> >>
> >
> >> I am working on a basic Samza task that pulls from one Kafka topic and writes to another. This task runs in Yarn but the Output topic does not contain any data.
> >
> >>
> >
> >>
> >
> >> In order to troubleshoot this more effectively I would like to log the incoming message as my example below. Ideally, I would like to be able to see the log messages in Yarn, maybe in the .out files in the /logs directory.
> >
> >>
> >
> >>
> >
> >> Any advice is appreciated.
> >
> >>
> >
> >>
> >
> >>
> >
> >> Here is my task:
> >
> >>
> >
> >>
> >
> >> package com.project.samza.tasks;
> >
> >>
> >
> >>
> >
> >> import java.util.Map;
> >
> >> import org.apache.samza.system.IncomingMessageEnvelope;
> >
> >> import org.apache.samza.system.OutgoingMessageEnvelope;
> >
> >> import org.apache.samza.system.SystemStream;
> >
> >> import org.apache.samza.task.MessageCollector;
> >
> >> import org.apache.samza.task.StreamTask;
> >
> >> import org.apache.samza.task.TaskCoordinator;
> >
> >>
> >
> >>
> >
> >> public class exampleStreamTask implements StreamTask {
> >
> >> private static final SystemStream OUTPUT_STREAM = new SystemStream(“kafka”, “new-topic-test”);
> >
> >>
> >
> >>
> >
> >> @Override
> >
> >> public void process(IncomingMessageEnvelope envelope,
> >
> >> MessageCollector collector,
> >
> >> TaskCoordinator coordinator) {
> >
> >> String msg = (String) envelope.getMessage();
> >
> >> System.out.println(msg);
> >
> >> collector.send(new OutgoingMessageEnvelope(OUTPUT_STREAM, msg));
> >
> >> }
> >
> >> }
> >
> >>
> >
> >>
> >
> >> Thanks,
> >
> >>
> >
> >>
> >
> >> Jason
>
Re: Best way to log from inside a Samza task?
Posted by ja...@marketingscience.co.
Rick,
>From your example, what is the correct way to call the logger in the task? Something like:
import java.util.Map;
import org.apache.samza.system.IncomingMessageEnvelope;
import org.apache.samza.system.OutgoingMessageEnvelope;
import org.apache.samza.system.SystemStream;
import org.apache.samza.task.MessageCollector;
import org.apache.samza.task.StreamTask;
import org.apache.samza.task.TaskCoordinator;
public class uimodPageloadStreamTask implements StreamTask {
private static final SystemStream OUTPUT_STREAM = new SystemStream(“kafka”, “kafka-output-test”);
@Override
public void process(IncomingMessageEnvelope envelope,
MessageCollector collector,
TaskCoordinator coordinator) {
logger.info(envelope.getMessage());
collector.send(new OutgoingMessageEnvelope(OUTPUT_STREAM, text));
}
}
Also, in what log file would the log messages end up? /logs/stdout for that application container?
I got System.out.println() to work similarly but it seems to be doing some formatting of the messages.
I am trying to consume messages from one Kafka topic and produce to another. While I can use zookeeper to see the messages in the originating topic they never make it to the destination and I am trying to find out why.
Thanks for your help,
Jason
On Friday, Jun 26, 2558 at 12:03, Rick Mangi <ri...@chartbeat.com>, wrote:
If you do something like this in your log4j.xml
<root>
<priority value="INFO" />
<appender-ref ref="RollingAppender" />
</root>
<logger name="cbsamza" additivity="false">
<level value=“DEBUG" />
<appender-ref ref="RollingAppender" />
</logger>
the root controls samza’s logging and the logger controls your own… I haven’t managed to get the imx configuration working yet.
> On Jun 26, 2015, at 1:50 PM, jason@marketingscience.co wrote:
>
> I was almost there. Got it now. Thanks for your help Rick.
>
>
>
>
> Cheers,
>
>
>
>
> Jason
>
>
>
>
>
>
>
>
>
> On Friday, Jun 26, 2558 at 11:43, Rick Mangi <ri...@chartbeat.com>, wrote:
> Hey Jason,
>
>
> If you configure log4j as described here: http://samza.apache.org/learn/documentation/0.9/jobs/logging.html <http://samza.apache.org/learn/documentation/0.9/jobs/logging.html>
>
>
> Your log statements will wind up in the samza-container logs which you can get to via the application master gui.
>
>
> hth,
>
>
> Rick
>
>
>
>> On Jun 26, 2015, at 1:39 PM, jason@marketingscience.co wrote:
>
>>
>
>> Hello,
>
>>
>
>>
>
>> I am working on a basic Samza task that pulls from one Kafka topic and writes to another. This task runs in Yarn but the Output topic does not contain any data.
>
>>
>
>>
>
>> In order to troubleshoot this more effectively I would like to log the incoming message as my example below. Ideally, I would like to be able to see the log messages in Yarn, maybe in the .out files in the /logs directory.
>
>>
>
>>
>
>> Any advice is appreciated.
>
>>
>
>>
>
>>
>
>> Here is my task:
>
>>
>
>>
>
>> package com.project.samza.tasks;
>
>>
>
>>
>
>> import java.util.Map;
>
>> import org.apache.samza.system.IncomingMessageEnvelope;
>
>> import org.apache.samza.system.OutgoingMessageEnvelope;
>
>> import org.apache.samza.system.SystemStream;
>
>> import org.apache.samza.task.MessageCollector;
>
>> import org.apache.samza.task.StreamTask;
>
>> import org.apache.samza.task.TaskCoordinator;
>
>>
>
>>
>
>> public class exampleStreamTask implements StreamTask {
>
>> private static final SystemStream OUTPUT_STREAM = new SystemStream(“kafka”, “new-topic-test”);
>
>>
>
>>
>
>> @Override
>
>> public void process(IncomingMessageEnvelope envelope,
>
>> MessageCollector collector,
>
>> TaskCoordinator coordinator) {
>
>> String msg = (String) envelope.getMessage();
>
>> System.out.println(msg);
>
>> collector.send(new OutgoingMessageEnvelope(OUTPUT_STREAM, msg));
>
>> }
>
>> }
>
>>
>
>>
>
>> Thanks,
>
>>
>
>>
>
>> Jason
Re: Best way to log from inside a Samza task?
Posted by Rick Mangi <ri...@chartbeat.com>.
If you do something like this in your log4j.xml
<root>
<priority value="INFO" />
<appender-ref ref="RollingAppender" />
</root>
<logger name="cbsamza" additivity="false">
<level value=“DEBUG" />
<appender-ref ref="RollingAppender" />
</logger>
the root controls samza’s logging and the logger controls your own… I haven’t managed to get the imx configuration working yet.
> On Jun 26, 2015, at 1:50 PM, jason@marketingscience.co wrote:
>
> I was almost there. Got it now. Thanks for your help Rick.
>
>
>
>
> Cheers,
>
>
>
>
> Jason
>
>
>
>
>
>
>
>
>
> On Friday, Jun 26, 2558 at 11:43, Rick Mangi <ri...@chartbeat.com>, wrote:
> Hey Jason,
>
>
> If you configure log4j as described here: http://samza.apache.org/learn/documentation/0.9/jobs/logging.html <http://samza.apache.org/learn/documentation/0.9/jobs/logging.html>
>
>
> Your log statements will wind up in the samza-container logs which you can get to via the application master gui.
>
>
> hth,
>
>
> Rick
>
>
>
>> On Jun 26, 2015, at 1:39 PM, jason@marketingscience.co wrote:
>
>>
>
>> Hello,
>
>>
>
>>
>
>> I am working on a basic Samza task that pulls from one Kafka topic and writes to another. This task runs in Yarn but the Output topic does not contain any data.
>
>>
>
>>
>
>> In order to troubleshoot this more effectively I would like to log the incoming message as my example below. Ideally, I would like to be able to see the log messages in Yarn, maybe in the .out files in the /logs directory.
>
>>
>
>>
>
>> Any advice is appreciated.
>
>>
>
>>
>
>>
>
>> Here is my task:
>
>>
>
>>
>
>> package com.project.samza.tasks;
>
>>
>
>>
>
>> import java.util.Map;
>
>> import org.apache.samza.system.IncomingMessageEnvelope;
>
>> import org.apache.samza.system.OutgoingMessageEnvelope;
>
>> import org.apache.samza.system.SystemStream;
>
>> import org.apache.samza.task.MessageCollector;
>
>> import org.apache.samza.task.StreamTask;
>
>> import org.apache.samza.task.TaskCoordinator;
>
>>
>
>>
>
>> public class exampleStreamTask implements StreamTask {
>
>> private static final SystemStream OUTPUT_STREAM = new SystemStream(“kafka”, “new-topic-test”);
>
>>
>
>>
>
>> @Override
>
>> public void process(IncomingMessageEnvelope envelope,
>
>> MessageCollector collector,
>
>> TaskCoordinator coordinator) {
>
>> String msg = (String) envelope.getMessage();
>
>> System.out.println(msg);
>
>> collector.send(new OutgoingMessageEnvelope(OUTPUT_STREAM, msg));
>
>> }
>
>> }
>
>>
>
>>
>
>> Thanks,
>
>>
>
>>
>
>> Jason
Re: Best way to log from inside a Samza task?
Posted by ja...@marketingscience.co.
I was almost there. Got it now. Thanks for your help Rick.
Cheers,
Jason
On Friday, Jun 26, 2558 at 11:43, Rick Mangi <ri...@chartbeat.com>, wrote:
Hey Jason,
If you configure log4j as described here: http://samza.apache.org/learn/documentation/0.9/jobs/logging.html <http://samza.apache.org/learn/documentation/0.9/jobs/logging.html>
Your log statements will wind up in the samza-container logs which you can get to via the application master gui.
hth,
Rick
> On Jun 26, 2015, at 1:39 PM, jason@marketingscience.co wrote:
>
> Hello,
>
>
> I am working on a basic Samza task that pulls from one Kafka topic and writes to another. This task runs in Yarn but the Output topic does not contain any data.
>
>
> In order to troubleshoot this more effectively I would like to log the incoming message as my example below. Ideally, I would like to be able to see the log messages in Yarn, maybe in the .out files in the /logs directory.
>
>
> Any advice is appreciated.
>
>
>
> Here is my task:
>
>
> package com.project.samza.tasks;
>
>
> import java.util.Map;
> import org.apache.samza.system.IncomingMessageEnvelope;
> import org.apache.samza.system.OutgoingMessageEnvelope;
> import org.apache.samza.system.SystemStream;
> import org.apache.samza.task.MessageCollector;
> import org.apache.samza.task.StreamTask;
> import org.apache.samza.task.TaskCoordinator;
>
>
> public class exampleStreamTask implements StreamTask {
> private static final SystemStream OUTPUT_STREAM = new SystemStream(“kafka”, “new-topic-test”);
>
>
> @Override
> public void process(IncomingMessageEnvelope envelope,
> MessageCollector collector,
> TaskCoordinator coordinator) {
> String msg = (String) envelope.getMessage();
> System.out.println(msg);
> collector.send(new OutgoingMessageEnvelope(OUTPUT_STREAM, msg));
> }
> }
>
>
> Thanks,
>
>
> Jason
Re: Best way to log from inside a Samza task?
Posted by Rick Mangi <ri...@chartbeat.com>.
Hey Jason,
If you configure log4j as described here: http://samza.apache.org/learn/documentation/0.9/jobs/logging.html <http://samza.apache.org/learn/documentation/0.9/jobs/logging.html>
Your log statements will wind up in the samza-container logs which you can get to via the application master gui.
hth,
Rick
> On Jun 26, 2015, at 1:39 PM, jason@marketingscience.co wrote:
>
> Hello,
>
>
> I am working on a basic Samza task that pulls from one Kafka topic and writes to another. This task runs in Yarn but the Output topic does not contain any data.
>
>
> In order to troubleshoot this more effectively I would like to log the incoming message as my example below. Ideally, I would like to be able to see the log messages in Yarn, maybe in the .out files in the /logs directory.
>
>
> Any advice is appreciated.
>
>
>
> Here is my task:
>
>
> package com.project.samza.tasks;
>
>
> import java.util.Map;
> import org.apache.samza.system.IncomingMessageEnvelope;
> import org.apache.samza.system.OutgoingMessageEnvelope;
> import org.apache.samza.system.SystemStream;
> import org.apache.samza.task.MessageCollector;
> import org.apache.samza.task.StreamTask;
> import org.apache.samza.task.TaskCoordinator;
>
>
> public class exampleStreamTask implements StreamTask {
> private static final SystemStream OUTPUT_STREAM = new SystemStream(“kafka”, “new-topic-test”);
>
>
> @Override
> public void process(IncomingMessageEnvelope envelope,
> MessageCollector collector,
> TaskCoordinator coordinator) {
> String msg = (String) envelope.getMessage();
> System.out.println(msg);
> collector.send(new OutgoingMessageEnvelope(OUTPUT_STREAM, msg));
> }
> }
>
>
> Thanks,
>
>
> Jason