how to get the detailed warn or error log message from pig udfs

[刘振川 <zh...@ipinyou.com>]

hey,guys,
   is there any way to log the detailed warn or error message from my udfs ?  Apache Pig version 0.8.1-cdh3u1  
   i wrote the udf warn like this:
   
IpRegion region = null;
try {
region = IPAnalyzeSingleton.getInstance().citySeek(ip);
} catch (Exception e) {
warn("invalid ip[" + ip + "],will return -1 .", PigWarning.UDF_WARNING_1);
return -1;
}
   
   after run the script with this command:  pig -4 /tmp/log4j.properties test.pig 

   only got the count numbers of the warns,but i meaned to get the detailed warn info.
   WARN main org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning UDF_WARNING_1 9261846 time(s). 

   the log4j configure as follows:
   
log4j.rootLogger=WARN, B

log4j.appender.B = org.apache.log4j.DailyRollingFileAppender
log4j.appender.B.File = /tmp/pig/logs/log.log
log4j.appender.B.DatePattern='.'yyyy-MM-dd
log4j.appender.B.Append = true
log4j.appender.B.Threshold = WARN
log4j.appender.B.layout = org.apache.log4j.PatternLayout
log4j.appender.B.layout.ConversionPattern = %p %t %c - %m%n




lucas 

Re: how to get the detailed warn or error log message from pig udfs

[Ruslan Al-Fakikh <me...@gmail.com>]

Hi Lucas,

It seems that you are using org.apache.pig.EvalFunc.warn(String, Enum)
which acts differently. Check the code or Javadocs. It works through Hadoop
counters I guess. You can use a regular log4j warnings or just
Sysout.out.println. But keep in mind that your UDF is implemented on a
remote node, not on your client node. Usually you can find its outputs when
you look at the WEB UI of the job, when you find the specific task's logs.
But dropping too much WARN's (say one WARN per row) can be really hard for
Hadoop logs, as there can be millions of rows, so Hadoop counters are good
for such things.

Ruslan



On Wed, Apr 17, 2013 at 12:27 PM, 刘振川 <zh...@ipinyou.com> wrote:

>
> hey,guys,
>    is there any way to log the detailed warn or error message from my udfs
> ?  Apache Pig version 0.8.1-cdh3u1
>    i wrote the udf warn like this:
>
> IpRegion region = null;
> try {
> region = IPAnalyzeSingleton.getInstance().citySeek(ip);
> } catch (Exception e) {
> warn("invalid ip[" + ip + "],will return -1 .", PigWarning.UDF_WARNING_1);
> return -1;
> }
>
>    after run the script with this command:  pig -4 /tmp/log4j.properties
> test.pig
>
>    only got the count numbers of the warns,but i meaned to get the
> detailed warn info.
>    WARN main
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Encountered Warning UDF_WARNING_1 9261846 time(s).
>
>    the log4j configure as follows:
>
> log4j.rootLogger=WARN, B
>
> log4j.appender.B = org.apache.log4j.DailyRollingFileAppender
> log4j.appender.B.File = /tmp/pig/logs/log.log
> log4j.appender.B.DatePattern='.'yyyy-MM-dd
> log4j.appender.B.Append = true
> log4j.appender.B.Threshold = WARN
> log4j.appender.B.layout = org.apache.log4j.PatternLayout
> log4j.appender.B.layout.ConversionPattern = %p %t %c - %m%n
>
>
>
>
> lucas