Log messages based on run-time tests

[Greg Harman <gh...@gmail.com>]

Is there a best practice on generating run-time log entries based on  
data values (perhaps in the form of adding a row to another relation  
in the script)?

answer = FOREACH foo GENERATE id, bar;  -- If bar == null then log a  
warning.

The best I can think of uses FILTER, but this seems inefficient since  
I have to apply a filter to ANSWER instead of flagging the bad value  
as it came up in GENERATE:

warnings = FILTER answer BY (bar == null);

Re: Log messages based on run-time tests

[Greg Harman <gh...@gmail.com>]

In my particular case, I'd prefer to have the "log" in another Pig  
relation (I'm driving Pig programatically) so I can pick the entries  
out and handle them appropriately. The UDF approach is a decent idea  
though - I can certainly use that to maintain the log on the side,  
assuming that the number of log messages is relatively small compared  
to the source data.

thanks!

On Aug 26, 2009, at 7:54 AM, Mridul Muralidharan wrote:

>
> By logging message, you mean to hdfs ? to stderr ? or to some other  
> source ? That being said, since I am not sure if I get the actual  
> usecase, but if I am not wrong cant you not do :
>
> --
> answer0 = FOREACH foo GENERATE id, bar;  -- If bar == null then log  
> a warning.
>
> answer = FOREACH answer0 GENERATE id, CHECK_NULL(bar);
>
> --
> where CHECK_NULL is a udf which returns the input as output, but  
> does the null check and logs the message you need ?
>
>
>
>
> Regards,
> Mridul
>
>
> Greg Harman wrote:
>> Is there a best practice on generating run-time log entries based  
>> on data values (perhaps in the form of adding a row to another  
>> relation in the script)?
>> answer = FOREACH foo GENERATE id, bar;  -- If bar == null then log  
>> a warning.
>> The best I can think of uses FILTER, but this seems inefficient  
>> since I have to apply a filter to ANSWER instead of flagging the  
>> bad value as it came up in GENERATE:
>> warnings = FILTER answer BY (bar == null);
>

Re: Log messages based on run-time tests

[Mridul Muralidharan <mr...@yahoo-inc.com>]

By logging message, you mean to hdfs ? to stderr ? or to some other 
source ? That being said, since I am not sure if I get the actual 
usecase, but if I am not wrong cant you not do :

--
answer0 = FOREACH foo GENERATE id, bar;  -- If bar == null then log a 
warning.

answer = FOREACH answer0 GENERATE id, CHECK_NULL(bar);

--
where CHECK_NULL is a udf which returns the input as output, but does 
the null check and logs the message you need ?




Regards,
Mridul


Greg Harman wrote:
> Is there a best practice on generating run-time log entries based on 
> data values (perhaps in the form of adding a row to another relation in 
> the script)?
> 
> answer = FOREACH foo GENERATE id, bar;  -- If bar == null then log a 
> warning.
> 
> The best I can think of uses FILTER, but this seems inefficient since I 
> have to apply a filter to ANSWER instead of flagging the bad value as it 
> came up in GENERATE:
> 
> warnings = FILTER answer BY (bar == null);
>