You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Eduard Skaley <e....@gmail.com> on 2012/12/25 13:33:39 UTC
Map Shuffle Bytes
Hello guys,
I need a counter for shuffled bytes to the mappers.
Is there existing one or should I define one myself ?
How can I implement such a counter?
Thank you and happy Christmas time,
Eduard
Re: Map Shuffle Bytes
Posted by Rishi Yadav <ri...@infoobjects.com>.
Shuffled bytes go to reducers not mappers and the inbuilt counter for that
is "Reduce Shuffle Bytes".
I am not sure how you can implement user defined java counter here because
this phase is in between map and reduce.
Thanks and Regards,
Rishi Yadav
(o) 408.988.2000x113 || (f) 408.716.2726
InfoObjects Inc || http://www.infoobjects.com *(Big Data Solutions)*
*INC 500 Fastest growing company in 2012 || 2011*
*Best Place to work in Bay Area 2012 - *SF Business Times and the Silicon
Valley / San Jose Business Journal
2041 Mission College Boulevard, #280 || Santa Clara, CA 95054
On Tue, Dec 25, 2012 at 4:33 AM, Eduard Skaley <e....@gmail.com> wrote:
> Hello guys,
>
> I need a counter for shuffled bytes to the mappers.
> Is there existing one or should I define one myself ?
> How can I implement such a counter?
>
> Thank you and happy Christmas time,
> Eduard
>
Re: Map Shuffle Bytes
Posted by Eduard Skaley <e....@gmail.com>.
For this I need to know where an inputsplit is located. And where a join
is computed. How can I do this programmatically ?
> This isn't called 'shuffle' (but rather a plain remote read) so your
> original question was confusing, thanks for clarifying!
>
> In that case, you could count the bytes coming in from the required
> record reader - for example a TextRecordReader uses a Long key that
> denotes current offset in file, which you could use as a simple,
> progressing counter of bytes read thus far.
>
> On Wed, Dec 26, 2012 at 5:16 PM, Eduard Skaley <e....@gmail.com> wrote:
>> Hi,
>>
>> I mean TO the mappers. I'm using the CompositeInputFormat for my application
>> to compute map-side joins.
>> I want to join two datasets A and B one is stored on node 1 and the other
>> one on node 2.
>> For example if the join will be computed on node 2 then the inputsplit of
>> the dataset which is stored on node 1 has to be transferred to node 2.
>> I want to count the bytes which are shuffled (transferred) TO the mapper of
>> node 2.
>>
>>> Hi,
>>>
>>> What do you mean by "shuffled bytes [to] the mappers"? If you mean
>>> "from", it is "Reduce shuffle bytes" you look for; otherwise, you may
>>> be looking for the per-map counter of "Map output bytes".
>>>
>>> Per-partition counters can be constructed on the user side if needed,
>>> by pre-computing the partition before emit (using the same
>>> partitioner) and counting up the bytes of your objects for its
>>> counter.
>>>
>>> On Tue, Dec 25, 2012 at 6:03 PM, Eduard Skaley <e....@gmail.com>
>>> wrote:
>>>> Hello guys,
>>>>
>>>> I need a counter for shuffled bytes to the mappers.
>>>> Is there existing one or should I define one myself ?
>>>> How can I implement such a counter?
>>>>
>>>> Thank you and happy Christmas time,
>>>> Eduard
>>>
>>>
>
>
Re: Map Shuffle Bytes
Posted by Eduard Skaley <e....@gmail.com>.
For this I need to know where an inputsplit is located. And where a join
is computed. How can I do this programmatically ?
> This isn't called 'shuffle' (but rather a plain remote read) so your
> original question was confusing, thanks for clarifying!
>
> In that case, you could count the bytes coming in from the required
> record reader - for example a TextRecordReader uses a Long key that
> denotes current offset in file, which you could use as a simple,
> progressing counter of bytes read thus far.
>
> On Wed, Dec 26, 2012 at 5:16 PM, Eduard Skaley <e....@gmail.com> wrote:
>> Hi,
>>
>> I mean TO the mappers. I'm using the CompositeInputFormat for my application
>> to compute map-side joins.
>> I want to join two datasets A and B one is stored on node 1 and the other
>> one on node 2.
>> For example if the join will be computed on node 2 then the inputsplit of
>> the dataset which is stored on node 1 has to be transferred to node 2.
>> I want to count the bytes which are shuffled (transferred) TO the mapper of
>> node 2.
>>
>>> Hi,
>>>
>>> What do you mean by "shuffled bytes [to] the mappers"? If you mean
>>> "from", it is "Reduce shuffle bytes" you look for; otherwise, you may
>>> be looking for the per-map counter of "Map output bytes".
>>>
>>> Per-partition counters can be constructed on the user side if needed,
>>> by pre-computing the partition before emit (using the same
>>> partitioner) and counting up the bytes of your objects for its
>>> counter.
>>>
>>> On Tue, Dec 25, 2012 at 6:03 PM, Eduard Skaley <e....@gmail.com>
>>> wrote:
>>>> Hello guys,
>>>>
>>>> I need a counter for shuffled bytes to the mappers.
>>>> Is there existing one or should I define one myself ?
>>>> How can I implement such a counter?
>>>>
>>>> Thank you and happy Christmas time,
>>>> Eduard
>>>
>>>
>
>
Re: Map Shuffle Bytes
Posted by Eduard Skaley <e....@gmail.com>.
For this I need to know where an inputsplit is located. And where a join
is computed. How can I do this programmatically ?
> This isn't called 'shuffle' (but rather a plain remote read) so your
> original question was confusing, thanks for clarifying!
>
> In that case, you could count the bytes coming in from the required
> record reader - for example a TextRecordReader uses a Long key that
> denotes current offset in file, which you could use as a simple,
> progressing counter of bytes read thus far.
>
> On Wed, Dec 26, 2012 at 5:16 PM, Eduard Skaley <e....@gmail.com> wrote:
>> Hi,
>>
>> I mean TO the mappers. I'm using the CompositeInputFormat for my application
>> to compute map-side joins.
>> I want to join two datasets A and B one is stored on node 1 and the other
>> one on node 2.
>> For example if the join will be computed on node 2 then the inputsplit of
>> the dataset which is stored on node 1 has to be transferred to node 2.
>> I want to count the bytes which are shuffled (transferred) TO the mapper of
>> node 2.
>>
>>> Hi,
>>>
>>> What do you mean by "shuffled bytes [to] the mappers"? If you mean
>>> "from", it is "Reduce shuffle bytes" you look for; otherwise, you may
>>> be looking for the per-map counter of "Map output bytes".
>>>
>>> Per-partition counters can be constructed on the user side if needed,
>>> by pre-computing the partition before emit (using the same
>>> partitioner) and counting up the bytes of your objects for its
>>> counter.
>>>
>>> On Tue, Dec 25, 2012 at 6:03 PM, Eduard Skaley <e....@gmail.com>
>>> wrote:
>>>> Hello guys,
>>>>
>>>> I need a counter for shuffled bytes to the mappers.
>>>> Is there existing one or should I define one myself ?
>>>> How can I implement such a counter?
>>>>
>>>> Thank you and happy Christmas time,
>>>> Eduard
>>>
>>>
>
>
Re: Map Shuffle Bytes
Posted by Eduard Skaley <e....@gmail.com>.
For this I need to know where an inputsplit is located. And where a join
is computed. How can I do this programmatically ?
> This isn't called 'shuffle' (but rather a plain remote read) so your
> original question was confusing, thanks for clarifying!
>
> In that case, you could count the bytes coming in from the required
> record reader - for example a TextRecordReader uses a Long key that
> denotes current offset in file, which you could use as a simple,
> progressing counter of bytes read thus far.
>
> On Wed, Dec 26, 2012 at 5:16 PM, Eduard Skaley <e....@gmail.com> wrote:
>> Hi,
>>
>> I mean TO the mappers. I'm using the CompositeInputFormat for my application
>> to compute map-side joins.
>> I want to join two datasets A and B one is stored on node 1 and the other
>> one on node 2.
>> For example if the join will be computed on node 2 then the inputsplit of
>> the dataset which is stored on node 1 has to be transferred to node 2.
>> I want to count the bytes which are shuffled (transferred) TO the mapper of
>> node 2.
>>
>>> Hi,
>>>
>>> What do you mean by "shuffled bytes [to] the mappers"? If you mean
>>> "from", it is "Reduce shuffle bytes" you look for; otherwise, you may
>>> be looking for the per-map counter of "Map output bytes".
>>>
>>> Per-partition counters can be constructed on the user side if needed,
>>> by pre-computing the partition before emit (using the same
>>> partitioner) and counting up the bytes of your objects for its
>>> counter.
>>>
>>> On Tue, Dec 25, 2012 at 6:03 PM, Eduard Skaley <e....@gmail.com>
>>> wrote:
>>>> Hello guys,
>>>>
>>>> I need a counter for shuffled bytes to the mappers.
>>>> Is there existing one or should I define one myself ?
>>>> How can I implement such a counter?
>>>>
>>>> Thank you and happy Christmas time,
>>>> Eduard
>>>
>>>
>
>
Re: Map Shuffle Bytes
Posted by Harsh J <ha...@cloudera.com>.
This isn't called 'shuffle' (but rather a plain remote read) so your
original question was confusing, thanks for clarifying!
In that case, you could count the bytes coming in from the required
record reader - for example a TextRecordReader uses a Long key that
denotes current offset in file, which you could use as a simple,
progressing counter of bytes read thus far.
On Wed, Dec 26, 2012 at 5:16 PM, Eduard Skaley <e....@gmail.com> wrote:
> Hi,
>
> I mean TO the mappers. I'm using the CompositeInputFormat for my application
> to compute map-side joins.
> I want to join two datasets A and B one is stored on node 1 and the other
> one on node 2.
> For example if the join will be computed on node 2 then the inputsplit of
> the dataset which is stored on node 1 has to be transferred to node 2.
> I want to count the bytes which are shuffled (transferred) TO the mapper of
> node 2.
>
>> Hi,
>>
>> What do you mean by "shuffled bytes [to] the mappers"? If you mean
>> "from", it is "Reduce shuffle bytes" you look for; otherwise, you may
>> be looking for the per-map counter of "Map output bytes".
>>
>> Per-partition counters can be constructed on the user side if needed,
>> by pre-computing the partition before emit (using the same
>> partitioner) and counting up the bytes of your objects for its
>> counter.
>>
>> On Tue, Dec 25, 2012 at 6:03 PM, Eduard Skaley <e....@gmail.com>
>> wrote:
>>>
>>> Hello guys,
>>>
>>> I need a counter for shuffled bytes to the mappers.
>>> Is there existing one or should I define one myself ?
>>> How can I implement such a counter?
>>>
>>> Thank you and happy Christmas time,
>>> Eduard
>>
>>
>>
>
--
Harsh J
Re: Map Shuffle Bytes
Posted by Harsh J <ha...@cloudera.com>.
This isn't called 'shuffle' (but rather a plain remote read) so your
original question was confusing, thanks for clarifying!
In that case, you could count the bytes coming in from the required
record reader - for example a TextRecordReader uses a Long key that
denotes current offset in file, which you could use as a simple,
progressing counter of bytes read thus far.
On Wed, Dec 26, 2012 at 5:16 PM, Eduard Skaley <e....@gmail.com> wrote:
> Hi,
>
> I mean TO the mappers. I'm using the CompositeInputFormat for my application
> to compute map-side joins.
> I want to join two datasets A and B one is stored on node 1 and the other
> one on node 2.
> For example if the join will be computed on node 2 then the inputsplit of
> the dataset which is stored on node 1 has to be transferred to node 2.
> I want to count the bytes which are shuffled (transferred) TO the mapper of
> node 2.
>
>> Hi,
>>
>> What do you mean by "shuffled bytes [to] the mappers"? If you mean
>> "from", it is "Reduce shuffle bytes" you look for; otherwise, you may
>> be looking for the per-map counter of "Map output bytes".
>>
>> Per-partition counters can be constructed on the user side if needed,
>> by pre-computing the partition before emit (using the same
>> partitioner) and counting up the bytes of your objects for its
>> counter.
>>
>> On Tue, Dec 25, 2012 at 6:03 PM, Eduard Skaley <e....@gmail.com>
>> wrote:
>>>
>>> Hello guys,
>>>
>>> I need a counter for shuffled bytes to the mappers.
>>> Is there existing one or should I define one myself ?
>>> How can I implement such a counter?
>>>
>>> Thank you and happy Christmas time,
>>> Eduard
>>
>>
>>
>
--
Harsh J
Re: Map Shuffle Bytes
Posted by Harsh J <ha...@cloudera.com>.
This isn't called 'shuffle' (but rather a plain remote read) so your
original question was confusing, thanks for clarifying!
In that case, you could count the bytes coming in from the required
record reader - for example a TextRecordReader uses a Long key that
denotes current offset in file, which you could use as a simple,
progressing counter of bytes read thus far.
On Wed, Dec 26, 2012 at 5:16 PM, Eduard Skaley <e....@gmail.com> wrote:
> Hi,
>
> I mean TO the mappers. I'm using the CompositeInputFormat for my application
> to compute map-side joins.
> I want to join two datasets A and B one is stored on node 1 and the other
> one on node 2.
> For example if the join will be computed on node 2 then the inputsplit of
> the dataset which is stored on node 1 has to be transferred to node 2.
> I want to count the bytes which are shuffled (transferred) TO the mapper of
> node 2.
>
>> Hi,
>>
>> What do you mean by "shuffled bytes [to] the mappers"? If you mean
>> "from", it is "Reduce shuffle bytes" you look for; otherwise, you may
>> be looking for the per-map counter of "Map output bytes".
>>
>> Per-partition counters can be constructed on the user side if needed,
>> by pre-computing the partition before emit (using the same
>> partitioner) and counting up the bytes of your objects for its
>> counter.
>>
>> On Tue, Dec 25, 2012 at 6:03 PM, Eduard Skaley <e....@gmail.com>
>> wrote:
>>>
>>> Hello guys,
>>>
>>> I need a counter for shuffled bytes to the mappers.
>>> Is there existing one or should I define one myself ?
>>> How can I implement such a counter?
>>>
>>> Thank you and happy Christmas time,
>>> Eduard
>>
>>
>>
>
--
Harsh J
Re: Map Shuffle Bytes
Posted by Harsh J <ha...@cloudera.com>.
This isn't called 'shuffle' (but rather a plain remote read) so your
original question was confusing, thanks for clarifying!
In that case, you could count the bytes coming in from the required
record reader - for example a TextRecordReader uses a Long key that
denotes current offset in file, which you could use as a simple,
progressing counter of bytes read thus far.
On Wed, Dec 26, 2012 at 5:16 PM, Eduard Skaley <e....@gmail.com> wrote:
> Hi,
>
> I mean TO the mappers. I'm using the CompositeInputFormat for my application
> to compute map-side joins.
> I want to join two datasets A and B one is stored on node 1 and the other
> one on node 2.
> For example if the join will be computed on node 2 then the inputsplit of
> the dataset which is stored on node 1 has to be transferred to node 2.
> I want to count the bytes which are shuffled (transferred) TO the mapper of
> node 2.
>
>> Hi,
>>
>> What do you mean by "shuffled bytes [to] the mappers"? If you mean
>> "from", it is "Reduce shuffle bytes" you look for; otherwise, you may
>> be looking for the per-map counter of "Map output bytes".
>>
>> Per-partition counters can be constructed on the user side if needed,
>> by pre-computing the partition before emit (using the same
>> partitioner) and counting up the bytes of your objects for its
>> counter.
>>
>> On Tue, Dec 25, 2012 at 6:03 PM, Eduard Skaley <e....@gmail.com>
>> wrote:
>>>
>>> Hello guys,
>>>
>>> I need a counter for shuffled bytes to the mappers.
>>> Is there existing one or should I define one myself ?
>>> How can I implement such a counter?
>>>
>>> Thank you and happy Christmas time,
>>> Eduard
>>
>>
>>
>
--
Harsh J
Re: Map Shuffle Bytes
Posted by Eduard Skaley <e....@gmail.com>.
Hi,
I mean TO the mappers. I'm using the CompositeInputFormat for my
application to compute map-side joins.
I want to join two datasets A and B one is stored on node 1 and the
other one on node 2.
For example if the join will be computed on node 2 then the inputsplit
of the dataset which is stored on node 1 has to be transferred to node 2.
I want to count the bytes which are shuffled (transferred) TO the mapper
of node 2.
> Hi,
>
> What do you mean by "shuffled bytes [to] the mappers"? If you mean
> "from", it is "Reduce shuffle bytes" you look for; otherwise, you may
> be looking for the per-map counter of "Map output bytes".
>
> Per-partition counters can be constructed on the user side if needed,
> by pre-computing the partition before emit (using the same
> partitioner) and counting up the bytes of your objects for its
> counter.
>
> On Tue, Dec 25, 2012 at 6:03 PM, Eduard Skaley <e....@gmail.com> wrote:
>> Hello guys,
>>
>> I need a counter for shuffled bytes to the mappers.
>> Is there existing one or should I define one myself ?
>> How can I implement such a counter?
>>
>> Thank you and happy Christmas time,
>> Eduard
>
>
Re: Map Shuffle Bytes
Posted by Eduard Skaley <e....@gmail.com>.
Hi,
I mean TO the mappers. I'm using the CompositeInputFormat for my
application to compute map-side joins.
I want to join two datasets A and B one is stored on node 1 and the
other one on node 2.
For example if the join will be computed on node 2 then the inputsplit
of the dataset which is stored on node 1 has to be transferred to node 2.
I want to count the bytes which are shuffled (transferred) TO the mapper
of node 2.
> Hi,
>
> What do you mean by "shuffled bytes [to] the mappers"? If you mean
> "from", it is "Reduce shuffle bytes" you look for; otherwise, you may
> be looking for the per-map counter of "Map output bytes".
>
> Per-partition counters can be constructed on the user side if needed,
> by pre-computing the partition before emit (using the same
> partitioner) and counting up the bytes of your objects for its
> counter.
>
> On Tue, Dec 25, 2012 at 6:03 PM, Eduard Skaley <e....@gmail.com> wrote:
>> Hello guys,
>>
>> I need a counter for shuffled bytes to the mappers.
>> Is there existing one or should I define one myself ?
>> How can I implement such a counter?
>>
>> Thank you and happy Christmas time,
>> Eduard
>
>
Re: Map Shuffle Bytes
Posted by Eduard Skaley <e....@gmail.com>.
Hi,
I mean TO the mappers. I'm using the CompositeInputFormat for my
application to compute map-side joins.
I want to join two datasets A and B one is stored on node 1 and the
other one on node 2.
For example if the join will be computed on node 2 then the inputsplit
of the dataset which is stored on node 1 has to be transferred to node 2.
I want to count the bytes which are shuffled (transferred) TO the mapper
of node 2.
> Hi,
>
> What do you mean by "shuffled bytes [to] the mappers"? If you mean
> "from", it is "Reduce shuffle bytes" you look for; otherwise, you may
> be looking for the per-map counter of "Map output bytes".
>
> Per-partition counters can be constructed on the user side if needed,
> by pre-computing the partition before emit (using the same
> partitioner) and counting up the bytes of your objects for its
> counter.
>
> On Tue, Dec 25, 2012 at 6:03 PM, Eduard Skaley <e....@gmail.com> wrote:
>> Hello guys,
>>
>> I need a counter for shuffled bytes to the mappers.
>> Is there existing one or should I define one myself ?
>> How can I implement such a counter?
>>
>> Thank you and happy Christmas time,
>> Eduard
>
>
Re: Map Shuffle Bytes
Posted by Eduard Skaley <e....@gmail.com>.
Hi,
I mean TO the mappers. I'm using the CompositeInputFormat for my
application to compute map-side joins.
I want to join two datasets A and B one is stored on node 1 and the
other one on node 2.
For example if the join will be computed on node 2 then the inputsplit
of the dataset which is stored on node 1 has to be transferred to node 2.
I want to count the bytes which are shuffled (transferred) TO the mapper
of node 2.
> Hi,
>
> What do you mean by "shuffled bytes [to] the mappers"? If you mean
> "from", it is "Reduce shuffle bytes" you look for; otherwise, you may
> be looking for the per-map counter of "Map output bytes".
>
> Per-partition counters can be constructed on the user side if needed,
> by pre-computing the partition before emit (using the same
> partitioner) and counting up the bytes of your objects for its
> counter.
>
> On Tue, Dec 25, 2012 at 6:03 PM, Eduard Skaley <e....@gmail.com> wrote:
>> Hello guys,
>>
>> I need a counter for shuffled bytes to the mappers.
>> Is there existing one or should I define one myself ?
>> How can I implement such a counter?
>>
>> Thank you and happy Christmas time,
>> Eduard
>
>
Re: Map Shuffle Bytes
Posted by Harsh J <ha...@cloudera.com>.
Hi,
What do you mean by "shuffled bytes [to] the mappers"? If you mean
"from", it is "Reduce shuffle bytes" you look for; otherwise, you may
be looking for the per-map counter of "Map output bytes".
Per-partition counters can be constructed on the user side if needed,
by pre-computing the partition before emit (using the same
partitioner) and counting up the bytes of your objects for its
counter.
On Tue, Dec 25, 2012 at 6:03 PM, Eduard Skaley <e....@gmail.com> wrote:
> Hello guys,
>
> I need a counter for shuffled bytes to the mappers.
> Is there existing one or should I define one myself ?
> How can I implement such a counter?
>
> Thank you and happy Christmas time,
> Eduard
--
Harsh J
Re: Map Shuffle Bytes
Posted by Rishi Yadav <ri...@infoobjects.com>.
Shuffled bytes go to reducers not mappers and the inbuilt counter for that
is "Reduce Shuffle Bytes".
I am not sure how you can implement user defined java counter here because
this phase is in between map and reduce.
Thanks and Regards,
Rishi Yadav
(o) 408.988.2000x113 || (f) 408.716.2726
InfoObjects Inc || http://www.infoobjects.com *(Big Data Solutions)*
*INC 500 Fastest growing company in 2012 || 2011*
*Best Place to work in Bay Area 2012 - *SF Business Times and the Silicon
Valley / San Jose Business Journal
2041 Mission College Boulevard, #280 || Santa Clara, CA 95054
On Tue, Dec 25, 2012 at 4:33 AM, Eduard Skaley <e....@gmail.com> wrote:
> Hello guys,
>
> I need a counter for shuffled bytes to the mappers.
> Is there existing one or should I define one myself ?
> How can I implement such a counter?
>
> Thank you and happy Christmas time,
> Eduard
>
Re: Map Shuffle Bytes
Posted by Rishi Yadav <ri...@infoobjects.com>.
Shuffled bytes go to reducers not mappers and the inbuilt counter for that
is "Reduce Shuffle Bytes".
I am not sure how you can implement user defined java counter here because
this phase is in between map and reduce.
Thanks and Regards,
Rishi Yadav
(o) 408.988.2000x113 || (f) 408.716.2726
InfoObjects Inc || http://www.infoobjects.com *(Big Data Solutions)*
*INC 500 Fastest growing company in 2012 || 2011*
*Best Place to work in Bay Area 2012 - *SF Business Times and the Silicon
Valley / San Jose Business Journal
2041 Mission College Boulevard, #280 || Santa Clara, CA 95054
On Tue, Dec 25, 2012 at 4:33 AM, Eduard Skaley <e....@gmail.com> wrote:
> Hello guys,
>
> I need a counter for shuffled bytes to the mappers.
> Is there existing one or should I define one myself ?
> How can I implement such a counter?
>
> Thank you and happy Christmas time,
> Eduard
>
Re: Map Shuffle Bytes
Posted by Rishi Yadav <ri...@infoobjects.com>.
Shuffled bytes go to reducers not mappers and the inbuilt counter for that
is "Reduce Shuffle Bytes".
I am not sure how you can implement user defined java counter here because
this phase is in between map and reduce.
Thanks and Regards,
Rishi Yadav
(o) 408.988.2000x113 || (f) 408.716.2726
InfoObjects Inc || http://www.infoobjects.com *(Big Data Solutions)*
*INC 500 Fastest growing company in 2012 || 2011*
*Best Place to work in Bay Area 2012 - *SF Business Times and the Silicon
Valley / San Jose Business Journal
2041 Mission College Boulevard, #280 || Santa Clara, CA 95054
On Tue, Dec 25, 2012 at 4:33 AM, Eduard Skaley <e....@gmail.com> wrote:
> Hello guys,
>
> I need a counter for shuffled bytes to the mappers.
> Is there existing one or should I define one myself ?
> How can I implement such a counter?
>
> Thank you and happy Christmas time,
> Eduard
>
Re: Map Shuffle Bytes
Posted by Harsh J <ha...@cloudera.com>.
Hi,
What do you mean by "shuffled bytes [to] the mappers"? If you mean
"from", it is "Reduce shuffle bytes" you look for; otherwise, you may
be looking for the per-map counter of "Map output bytes".
Per-partition counters can be constructed on the user side if needed,
by pre-computing the partition before emit (using the same
partitioner) and counting up the bytes of your objects for its
counter.
On Tue, Dec 25, 2012 at 6:03 PM, Eduard Skaley <e....@gmail.com> wrote:
> Hello guys,
>
> I need a counter for shuffled bytes to the mappers.
> Is there existing one or should I define one myself ?
> How can I implement such a counter?
>
> Thank you and happy Christmas time,
> Eduard
--
Harsh J
Re: Map Shuffle Bytes
Posted by Harsh J <ha...@cloudera.com>.
Hi,
What do you mean by "shuffled bytes [to] the mappers"? If you mean
"from", it is "Reduce shuffle bytes" you look for; otherwise, you may
be looking for the per-map counter of "Map output bytes".
Per-partition counters can be constructed on the user side if needed,
by pre-computing the partition before emit (using the same
partitioner) and counting up the bytes of your objects for its
counter.
On Tue, Dec 25, 2012 at 6:03 PM, Eduard Skaley <e....@gmail.com> wrote:
> Hello guys,
>
> I need a counter for shuffled bytes to the mappers.
> Is there existing one or should I define one myself ?
> How can I implement such a counter?
>
> Thank you and happy Christmas time,
> Eduard
--
Harsh J
Re: Map Shuffle Bytes
Posted by Harsh J <ha...@cloudera.com>.
Hi,
What do you mean by "shuffled bytes [to] the mappers"? If you mean
"from", it is "Reduce shuffle bytes" you look for; otherwise, you may
be looking for the per-map counter of "Map output bytes".
Per-partition counters can be constructed on the user side if needed,
by pre-computing the partition before emit (using the same
partitioner) and counting up the bytes of your objects for its
counter.
On Tue, Dec 25, 2012 at 6:03 PM, Eduard Skaley <e....@gmail.com> wrote:
> Hello guys,
>
> I need a counter for shuffled bytes to the mappers.
> Is there existing one or should I define one myself ?
> How can I implement such a counter?
>
> Thank you and happy Christmas time,
> Eduard
--
Harsh J