You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Thejas M Nair (JIRA)" <ji...@apache.org> on 2011/08/06 00:36:27 UTC
[jira] [Created] (PIG-2207) Support custom counters for aggregating
warnings from different udfs
Support custom counters for aggregating warnings from different udfs
--------------------------------------------------------------------
Key: PIG-2207
URL: https://issues.apache.org/jira/browse/PIG-2207
Project: Pig
Issue Type: Improvement
Reporter: Thejas M Nair
Fix For: 0.10
Pig allows udfs to aggregate warning messages instead of writing out a separate warning message each time. Udfs can do this by logging the warning using EvalFunc.warn(String msg, Enum) call. But the udfs are forced to use PigWarning class if the warning needs to be printed at the end of the pig script .
For example, with the changes in PIG-2191, some of the builtin udfs are using PigWarning.UDF_WARNING_1 as argument in calls to EvalFunc.warn. This will result in the warning count being printed on STDERR -
{code}
2011-08-05 22:10:29,285 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning UDF_WARNING_1 2 time(s).
2011-08-05 22:10:29,285 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
{code}
But it would be better if a udf such as the LOWER udf could use a custom warning counter, and the STDERR is like -
{code}
2011-08-05 22:10:29,285 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning LOWER_FUNC_INPUT_WARNING 2 time(s).
2011-08-05 22:10:29,285 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
{code}
A new function could be added to support this - (something like) EvalFunc.warn(String warnName, String warnMsg); A specific counter group could be used for udf warnings (see org.apache.hadoop.mapred.Counters), and counters for that group could be done during final warning aggregation in done in MapReduceLauncher.computeWarningAggregate().
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2207) Support custom counters for
aggregating warnings from different udfs
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082003#comment-13082003 ]
Thejas M Nair commented on PIG-2207:
------------------------------------
It will not be a good idea for udfs to use separate counters until the next-gen mapreduce is in (MR-279). A large number of counters can cause the the job to be killed, for hadoop 0.20.x (see MAPREDUCE-1943). MR-279 will enable much larger number of counters to be used.
> Support custom counters for aggregating warnings from different udfs
> --------------------------------------------------------------------
>
> Key: PIG-2207
> URL: https://issues.apache.org/jira/browse/PIG-2207
> Project: Pig
> Issue Type: Improvement
> Reporter: Thejas M Nair
> Labels: newbie
> Fix For: 0.10
>
>
> Pig allows udfs to aggregate warning messages instead of writing out a separate warning message each time. Udfs can do this by logging the warning using EvalFunc.warn(String msg, Enum) call. But the udfs are forced to use PigWarning class if the warning needs to be printed at the end of the pig script .
> For example, with the changes in PIG-2191, some of the builtin udfs are using PigWarning.UDF_WARNING_1 as argument in calls to EvalFunc.warn. This will result in the warning count being printed on STDERR -
> {code}
> 2011-08-05 22:10:29,285 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning UDF_WARNING_1 2 time(s).
> 2011-08-05 22:10:29,285 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
> {code}
> But it would be better if a udf such as the LOWER udf could use a custom warning counter, and the STDERR is like -
> {code}
> 2011-08-05 22:10:29,285 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning LOWER_FUNC_INPUT_WARNING 2 time(s).
> 2011-08-05 22:10:29,285 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
> {code}
> A new function could be added to support this - (something like) EvalFunc.warn(String warnName, String warnMsg); A specific counter group could be used for udf warnings (see org.apache.hadoop.mapred.Counters), and counters for that group could be done during final warning aggregation in done in MapReduceLauncher.computeWarningAggregate().
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2207) Support custom counters for
aggregating warnings from different udfs
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080280#comment-13080280 ]
Dmitriy V. Ryaboy commented on PIG-2207:
----------------------------------------
You read my mind.
I think for starters, we should increment the warning counters in groups based on the UDF. Right now all the UDF_WARNING_1s roll together, but we could split them into MyUDF / UDF_WARNING_1, MyOtherUDF / UDF_WARNING_2
I also think the warning message should be printed "at least once" -- just the counters alone aren't always sufficient. We could get this fairly cheaply by keeping a weakref hashmap of seen warnings, and logging the message every time we see a new warning.
> Support custom counters for aggregating warnings from different udfs
> --------------------------------------------------------------------
>
> Key: PIG-2207
> URL: https://issues.apache.org/jira/browse/PIG-2207
> Project: Pig
> Issue Type: Improvement
> Reporter: Thejas M Nair
> Labels: newbie
> Fix For: 0.10
>
>
> Pig allows udfs to aggregate warning messages instead of writing out a separate warning message each time. Udfs can do this by logging the warning using EvalFunc.warn(String msg, Enum) call. But the udfs are forced to use PigWarning class if the warning needs to be printed at the end of the pig script .
> For example, with the changes in PIG-2191, some of the builtin udfs are using PigWarning.UDF_WARNING_1 as argument in calls to EvalFunc.warn. This will result in the warning count being printed on STDERR -
> {code}
> 2011-08-05 22:10:29,285 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning UDF_WARNING_1 2 time(s).
> 2011-08-05 22:10:29,285 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
> {code}
> But it would be better if a udf such as the LOWER udf could use a custom warning counter, and the STDERR is like -
> {code}
> 2011-08-05 22:10:29,285 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning LOWER_FUNC_INPUT_WARNING 2 time(s).
> 2011-08-05 22:10:29,285 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
> {code}
> A new function could be added to support this - (something like) EvalFunc.warn(String warnName, String warnMsg); A specific counter group could be used for udf warnings (see org.apache.hadoop.mapred.Counters), and counters for that group could be done during final warning aggregation in done in MapReduceLauncher.computeWarningAggregate().
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2207) Support custom counters for aggregating
warnings from different udfs
Posted by "Olga Natkovich (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-2207:
--------------------------------
Fix Version/s: (was: 0.10)
> Support custom counters for aggregating warnings from different udfs
> --------------------------------------------------------------------
>
> Key: PIG-2207
> URL: https://issues.apache.org/jira/browse/PIG-2207
> Project: Pig
> Issue Type: Improvement
> Reporter: Thejas M Nair
> Labels: newbie
>
> Pig allows udfs to aggregate warning messages instead of writing out a separate warning message each time. Udfs can do this by logging the warning using EvalFunc.warn(String msg, Enum) call. But the udfs are forced to use PigWarning class if the warning needs to be printed at the end of the pig script .
> For example, with the changes in PIG-2191, some of the builtin udfs are using PigWarning.UDF_WARNING_1 as argument in calls to EvalFunc.warn. This will result in the warning count being printed on STDERR -
> {code}
> 2011-08-05 22:10:29,285 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning UDF_WARNING_1 2 time(s).
> 2011-08-05 22:10:29,285 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
> {code}
> But it would be better if a udf such as the LOWER udf could use a custom warning counter, and the STDERR is like -
> {code}
> 2011-08-05 22:10:29,285 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning LOWER_FUNC_INPUT_WARNING 2 time(s).
> 2011-08-05 22:10:29,285 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
> {code}
> A new function could be added to support this - (something like) EvalFunc.warn(String warnName, String warnMsg); A specific counter group could be used for udf warnings (see org.apache.hadoop.mapred.Counters), and counters for that group could be done during final warning aggregation in done in MapReduceLauncher.computeWarningAggregate().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2207) Support custom counters for
aggregating warnings from different udfs
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082638#comment-13082638 ]
Dmitriy V. Ryaboy commented on PIG-2207:
----------------------------------------
Right, the client communication is done via counters, and if they want to debug, they can go look in the logs. Other than syntax errors, all of my Pig job debugging starts with looking at task logs anyway.
I see what you meant with the counter names now -- so the counter would be "PigCounters.UDF_WARNING_1-dereferencing_null_in_LOWER" for example. This accomplishes the goal of communicating the error to the client, but still produces many counters, which you just said goes against the recommendation in MAPREDUCE-1943, and also makes the counter name too long -- 1943 restricts counter names to 64 chars.
What is the value of keeping all udf counters in the same group?
> Support custom counters for aggregating warnings from different udfs
> --------------------------------------------------------------------
>
> Key: PIG-2207
> URL: https://issues.apache.org/jira/browse/PIG-2207
> Project: Pig
> Issue Type: Improvement
> Reporter: Thejas M Nair
> Labels: newbie
> Fix For: 0.10
>
>
> Pig allows udfs to aggregate warning messages instead of writing out a separate warning message each time. Udfs can do this by logging the warning using EvalFunc.warn(String msg, Enum) call. But the udfs are forced to use PigWarning class if the warning needs to be printed at the end of the pig script .
> For example, with the changes in PIG-2191, some of the builtin udfs are using PigWarning.UDF_WARNING_1 as argument in calls to EvalFunc.warn. This will result in the warning count being printed on STDERR -
> {code}
> 2011-08-05 22:10:29,285 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning UDF_WARNING_1 2 time(s).
> 2011-08-05 22:10:29,285 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
> {code}
> But it would be better if a udf such as the LOWER udf could use a custom warning counter, and the STDERR is like -
> {code}
> 2011-08-05 22:10:29,285 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning LOWER_FUNC_INPUT_WARNING 2 time(s).
> 2011-08-05 22:10:29,285 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
> {code}
> A new function could be added to support this - (something like) EvalFunc.warn(String warnName, String warnMsg); A specific counter group could be used for udf warnings (see org.apache.hadoop.mapred.Counters), and counters for that group could be done during final warning aggregation in done in MapReduceLauncher.computeWarningAggregate().
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2207) Support custom counters for
aggregating warnings from different udfs
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082556#comment-13082556 ]
Thejas M Nair commented on PIG-2207:
------------------------------------
bq. I also think the warning message should be printed "at least once" – just the counters alone aren't always sufficient. We could get this fairly cheaply by keeping a weakref hashmap of seen warnings, and logging the message every time we see a new warning.
This way the single line of warning can be printed in the backend (task) logs, but not in client to the user. It does not provide a means for the backend to client communication.
bq. Your proposal suggests custom counters via strings, so we have this problem either way. At least making the group match the class that issues the warning constrains the space of possible counters to num_udfs x 4_generic_warnings, whereas the space of possible strings is (for our purposes) unlimited.
I think the counter name should based on warnName + *first* warnMsg, in EvalFunc.warn(String warnName, String warnMsg). That way a single warning message also will be displayed on client side, when warning aggregations is turned on. When warning aggregation is turned off, each warnMsg goes into log.
> Support custom counters for aggregating warnings from different udfs
> --------------------------------------------------------------------
>
> Key: PIG-2207
> URL: https://issues.apache.org/jira/browse/PIG-2207
> Project: Pig
> Issue Type: Improvement
> Reporter: Thejas M Nair
> Labels: newbie
> Fix For: 0.10
>
>
> Pig allows udfs to aggregate warning messages instead of writing out a separate warning message each time. Udfs can do this by logging the warning using EvalFunc.warn(String msg, Enum) call. But the udfs are forced to use PigWarning class if the warning needs to be printed at the end of the pig script .
> For example, with the changes in PIG-2191, some of the builtin udfs are using PigWarning.UDF_WARNING_1 as argument in calls to EvalFunc.warn. This will result in the warning count being printed on STDERR -
> {code}
> 2011-08-05 22:10:29,285 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning UDF_WARNING_1 2 time(s).
> 2011-08-05 22:10:29,285 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
> {code}
> But it would be better if a udf such as the LOWER udf could use a custom warning counter, and the STDERR is like -
> {code}
> 2011-08-05 22:10:29,285 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning LOWER_FUNC_INPUT_WARNING 2 time(s).
> 2011-08-05 22:10:29,285 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
> {code}
> A new function could be added to support this - (something like) EvalFunc.warn(String warnName, String warnMsg); A specific counter group could be used for udf warnings (see org.apache.hadoop.mapred.Counters), and counters for that group could be done during final warning aggregation in done in MapReduceLauncher.computeWarningAggregate().
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2207) Support custom counters for
aggregating warnings from different udfs
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082661#comment-13082661 ]
Thejas M Nair commented on PIG-2207:
------------------------------------
I think for now, it makes sense to go with what Dmitriy proposed in his first comment .
> Support custom counters for aggregating warnings from different udfs
> --------------------------------------------------------------------
>
> Key: PIG-2207
> URL: https://issues.apache.org/jira/browse/PIG-2207
> Project: Pig
> Issue Type: Improvement
> Reporter: Thejas M Nair
> Labels: newbie
> Fix For: 0.10
>
>
> Pig allows udfs to aggregate warning messages instead of writing out a separate warning message each time. Udfs can do this by logging the warning using EvalFunc.warn(String msg, Enum) call. But the udfs are forced to use PigWarning class if the warning needs to be printed at the end of the pig script .
> For example, with the changes in PIG-2191, some of the builtin udfs are using PigWarning.UDF_WARNING_1 as argument in calls to EvalFunc.warn. This will result in the warning count being printed on STDERR -
> {code}
> 2011-08-05 22:10:29,285 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning UDF_WARNING_1 2 time(s).
> 2011-08-05 22:10:29,285 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
> {code}
> But it would be better if a udf such as the LOWER udf could use a custom warning counter, and the STDERR is like -
> {code}
> 2011-08-05 22:10:29,285 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning LOWER_FUNC_INPUT_WARNING 2 time(s).
> 2011-08-05 22:10:29,285 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
> {code}
> A new function could be added to support this - (something like) EvalFunc.warn(String warnName, String warnMsg); A specific counter group could be used for udf warnings (see org.apache.hadoop.mapred.Counters), and counters for that group could be done during final warning aggregation in done in MapReduceLauncher.computeWarningAggregate().
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2207) Support custom counters for
aggregating warnings from different udfs
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082658#comment-13082658 ]
Thejas M Nair commented on PIG-2207:
------------------------------------
bq. What is the value of keeping all udf counters in the same group?
On the client side that will help finding which counters are warning counters, so that those can be printed.
bq. Other than syntax errors, all of my Pig job debugging starts with looking at task logs anyway.
I was thinking of the case where you don't know that you are supposed to start debugging :) . I guess, the generic udf warnings counter could might be good enough for that.
The 64 char limit would be another reason why what I proposed will not work with MAPREDUCE-1943. But even MR-279 might also end up having such limitations in the future. (Unless pig ships with a custom MR application master!)
> Support custom counters for aggregating warnings from different udfs
> --------------------------------------------------------------------
>
> Key: PIG-2207
> URL: https://issues.apache.org/jira/browse/PIG-2207
> Project: Pig
> Issue Type: Improvement
> Reporter: Thejas M Nair
> Labels: newbie
> Fix For: 0.10
>
>
> Pig allows udfs to aggregate warning messages instead of writing out a separate warning message each time. Udfs can do this by logging the warning using EvalFunc.warn(String msg, Enum) call. But the udfs are forced to use PigWarning class if the warning needs to be printed at the end of the pig script .
> For example, with the changes in PIG-2191, some of the builtin udfs are using PigWarning.UDF_WARNING_1 as argument in calls to EvalFunc.warn. This will result in the warning count being printed on STDERR -
> {code}
> 2011-08-05 22:10:29,285 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning UDF_WARNING_1 2 time(s).
> 2011-08-05 22:10:29,285 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
> {code}
> But it would be better if a udf such as the LOWER udf could use a custom warning counter, and the STDERR is like -
> {code}
> 2011-08-05 22:10:29,285 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning LOWER_FUNC_INPUT_WARNING 2 time(s).
> 2011-08-05 22:10:29,285 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
> {code}
> A new function could be added to support this - (something like) EvalFunc.warn(String warnName, String warnMsg); A specific counter group could be used for udf warnings (see org.apache.hadoop.mapred.Counters), and counters for that group could be done during final warning aggregation in done in MapReduceLauncher.computeWarningAggregate().
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (PIG-2207) Support custom counters for
aggregating warnings from different udfs
Posted by "Timothy Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Timothy Chen reassigned PIG-2207:
---------------------------------
Assignee: Timothy Chen
> Support custom counters for aggregating warnings from different udfs
> --------------------------------------------------------------------
>
> Key: PIG-2207
> URL: https://issues.apache.org/jira/browse/PIG-2207
> Project: Pig
> Issue Type: Improvement
> Reporter: Thejas M Nair
> Assignee: Timothy Chen
> Labels: newbie
>
> Pig allows udfs to aggregate warning messages instead of writing out a separate warning message each time. Udfs can do this by logging the warning using EvalFunc.warn(String msg, Enum) call. But the udfs are forced to use PigWarning class if the warning needs to be printed at the end of the pig script .
> For example, with the changes in PIG-2191, some of the builtin udfs are using PigWarning.UDF_WARNING_1 as argument in calls to EvalFunc.warn. This will result in the warning count being printed on STDERR -
> {code}
> 2011-08-05 22:10:29,285 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning UDF_WARNING_1 2 time(s).
> 2011-08-05 22:10:29,285 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
> {code}
> But it would be better if a udf such as the LOWER udf could use a custom warning counter, and the STDERR is like -
> {code}
> 2011-08-05 22:10:29,285 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning LOWER_FUNC_INPUT_WARNING 2 time(s).
> 2011-08-05 22:10:29,285 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
> {code}
> A new function could be added to support this - (something like) EvalFunc.warn(String warnName, String warnMsg); A specific counter group could be used for udf warnings (see org.apache.hadoop.mapred.Counters), and counters for that group could be done during final warning aggregation in done in MapReduceLauncher.computeWarningAggregate().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2207) Support custom counters for
aggregating warnings from different udfs
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082137#comment-13082137 ]
Dmitriy V. Ryaboy commented on PIG-2207:
----------------------------------------
Your proposal suggests custom counters via strings, so we have this problem either way. At least making the group match the class that issues the warning constrains the space of possible counters to num_udfs x 4_generic_warnings, whereas the space of possible strings is (for our purposes) unlimited.
> Support custom counters for aggregating warnings from different udfs
> --------------------------------------------------------------------
>
> Key: PIG-2207
> URL: https://issues.apache.org/jira/browse/PIG-2207
> Project: Pig
> Issue Type: Improvement
> Reporter: Thejas M Nair
> Labels: newbie
> Fix For: 0.10
>
>
> Pig allows udfs to aggregate warning messages instead of writing out a separate warning message each time. Udfs can do this by logging the warning using EvalFunc.warn(String msg, Enum) call. But the udfs are forced to use PigWarning class if the warning needs to be printed at the end of the pig script .
> For example, with the changes in PIG-2191, some of the builtin udfs are using PigWarning.UDF_WARNING_1 as argument in calls to EvalFunc.warn. This will result in the warning count being printed on STDERR -
> {code}
> 2011-08-05 22:10:29,285 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning UDF_WARNING_1 2 time(s).
> 2011-08-05 22:10:29,285 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
> {code}
> But it would be better if a udf such as the LOWER udf could use a custom warning counter, and the STDERR is like -
> {code}
> 2011-08-05 22:10:29,285 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning LOWER_FUNC_INPUT_WARNING 2 time(s).
> 2011-08-05 22:10:29,285 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
> {code}
> A new function could be added to support this - (something like) EvalFunc.warn(String warnName, String warnMsg); A specific counter group could be used for udf warnings (see org.apache.hadoop.mapred.Counters), and counters for that group could be done during final warning aggregation in done in MapReduceLauncher.computeWarningAggregate().
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira