You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Corbin Hoenes <co...@tynt.com> on 2010/06/16 20:52:29 UTC
little help on reading debug output
Is there any documentation on how to read this output when I 'set debug on' I get in my reducer syslog:
DEBUG: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce - New For Each(true,true)[tuple] - 1-770
| |
| POBinCond[bag] - 1-768
| |
| |---Project[bag][1] - 1-764
| |
| |---POUserFunc(org.apache.pig.builtin.IsEmpty)[boolean] - 1-766
| | |
| | |---Project[bag][1] - 1-765
| |
| |---Constant({()}) - 1-767
| |
| Project[bag][2] - 1-769
DEBUG: org.apache.pig.data.InternalCachedBag - Memory can hold 45450 records, put the rest in spill file.
DEBUG: org.apache.pig.data.InternalCachedBag - Memory can hold 45192 records, put the rest in spill file.
DEBUG: org.apache.pig.data.InternalCachedBag - Memory can hold 44852 records, put the rest in spill file
Specifically what do the 1-7** numbers mean? Is it possible to get line numbers from the pig script :)
Also strange is that it seems that POUserFunc is telling me we are running the IsEmpty UDF but that UDF isn't being called in this script at all...is it possible pig is using it under the covers?
Re: little help on reading debug output
Posted by Daniel Dai <ji...@yahoo-inc.com>.
1-*** is the unique id assigned to each node. There is no line number,
but in trunk code, you will see alias in front of the node.
We sometime add UDF to implement specific Pig functions. For example, we
will add IsEmpty to implement outer join.
Daniel
Corbin Hoenes wrote:
> Is there any documentation on how to read this output when I 'set debug on' I get in my reducer syslog:
>
> DEBUG: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce - New For Each(true,true)[tuple] - 1-770
> | |
> | POBinCond[bag] - 1-768
> | |
> | |---Project[bag][1] - 1-764
> | |
> | |---POUserFunc(org.apache.pig.builtin.IsEmpty)[boolean] - 1-766
> | | |
> | | |---Project[bag][1] - 1-765
> | |
> | |---Constant({()}) - 1-767
> | |
> | Project[bag][2] - 1-769
> DEBUG: org.apache.pig.data.InternalCachedBag - Memory can hold 45450 records, put the rest in spill file.
> DEBUG: org.apache.pig.data.InternalCachedBag - Memory can hold 45192 records, put the rest in spill file.
> DEBUG: org.apache.pig.data.InternalCachedBag - Memory can hold 44852 records, put the rest in spill file
>
> Specifically what do the 1-7** numbers mean? Is it possible to get line numbers from the pig script :)
> Also strange is that it seems that POUserFunc is telling me we are running the IsEmpty UDF but that UDF isn't being called in this script at all...is it possible pig is using it under the covers?
>
>