You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Shen LI <ge...@gmail.com> on 2011/06/10 18:19:07 UTC
How many bytes does heart beat message and return value contain?
Hi All,
Does anyone know how many bytes does one heart beat message contain? And
what about the return value of heart beat message?
Thanks,
Shen
Re: How many bytes does heart beat message and return value contain?
Posted by Harsh J <ha...@cloudera.com>.
Shen,
Perhaps you could write some tests in Java to measure object sizes; or
you can run a profiler and measure it live (VisualVM lets you profile
with visuals)?
As Arun mentioned, it is really tough to say since every heartbeat
would carry variable sizes (I'd just provided empty-ish values)
depending on each of the Task's current, reported status and the
commands being issued back. But I believe you could form a formula of
sorts with careful-enough study.
On Sun, Jun 12, 2011 at 6:17 AM, Shen LI <ge...@gmail.com> wrote:
> Hi Harsh,
> Thank you very much for your reply. Do you know if there is any reference
> about calculating heart beat message size that I can turn to? If I have 4
> living task under each TaskTracker, how many bytes will one heart beat
> message contain?
> Thanks,
> Shen
>
> On Fri, Jun 10, 2011 at 5:06 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> Shen,
>>
>> It would really vary depending on how many TaskStatus objects the
>> TaskTracker has to report in the heartbeat. If there are none (zero
>> activity), I think the byte value (per Visual VM here) comes to about
>> 60B which you can perhaps add upon.
>>
>> Note that I'm using OSX so perhaps the ResourceStats carried within
>> aren't properly for me (some of these are Linux specific). It
>> shouldn't exceed 100B in any case though, I think.
>>
>> Similarly, the response carries variable numbers of TaskTrackerAction
>> objects, and other job related information. At zero activity I see it
>> reported as 32B.
>>
>> HTH. These are in memory values -- network tx. is made with some form
>> of serialization applied, and I'm unsure as to how much the variance
>> would be.
>>
>> On Fri, Jun 10, 2011 at 9:49 PM, Shen LI <ge...@gmail.com> wrote:
>> > Hi All,
>> > Does anyone know how many bytes does one heart beat message contain? And
>> > what about the return value of heart beat message?
>> > Thanks,
>> > Shen
>>
>>
>>
>> --
>> Harsh J
>
>
--
Harsh J
Re: How many bytes does heart beat message and return value contain?
Posted by Shen LI <ge...@gmail.com>.
Hi Harsh,
Thank you very much for your reply. Do you know if there is any reference
about calculating heart beat message size that I can turn to? If I have 4
living task under each TaskTracker, how many bytes will one heart beat
message contain?
Thanks,
Shen
On Fri, Jun 10, 2011 at 5:06 PM, Harsh J <ha...@cloudera.com> wrote:
> Shen,
>
> It would really vary depending on how many TaskStatus objects the
> TaskTracker has to report in the heartbeat. If there are none (zero
> activity), I think the byte value (per Visual VM here) comes to about
> 60B which you can perhaps add upon.
>
> Note that I'm using OSX so perhaps the ResourceStats carried within
> aren't properly for me (some of these are Linux specific). It
> shouldn't exceed 100B in any case though, I think.
>
> Similarly, the response carries variable numbers of TaskTrackerAction
> objects, and other job related information. At zero activity I see it
> reported as 32B.
>
> HTH. These are in memory values -- network tx. is made with some form
> of serialization applied, and I'm unsure as to how much the variance
> would be.
>
> On Fri, Jun 10, 2011 at 9:49 PM, Shen LI <ge...@gmail.com> wrote:
> > Hi All,
> > Does anyone know how many bytes does one heart beat message contain? And
> > what about the return value of heart beat message?
> > Thanks,
> > Shen
>
>
>
> --
> Harsh J
>
Re: How many bytes does heart beat message and return value contain?
Posted by Harsh J <ha...@cloudera.com>.
Shen,
It would really vary depending on how many TaskStatus objects the
TaskTracker has to report in the heartbeat. If there are none (zero
activity), I think the byte value (per Visual VM here) comes to about
60B which you can perhaps add upon.
Note that I'm using OSX so perhaps the ResourceStats carried within
aren't properly for me (some of these are Linux specific). It
shouldn't exceed 100B in any case though, I think.
Similarly, the response carries variable numbers of TaskTrackerAction
objects, and other job related information. At zero activity I see it
reported as 32B.
HTH. These are in memory values -- network tx. is made with some form
of serialization applied, and I'm unsure as to how much the variance
would be.
On Fri, Jun 10, 2011 at 9:49 PM, Shen LI <ge...@gmail.com> wrote:
> Hi All,
> Does anyone know how many bytes does one heart beat message contain? And
> what about the return value of heart beat message?
> Thanks,
> Shen
--
Harsh J
Re: How many bytes does heart beat message and return value contain?
Posted by Arun C Murthy <ac...@yahoo-inc.com>.
It really depends... the problem is that the TT's msg includes user-
counters.
So, in bad cases it could be infinite. hadoop-0.20.203 limits
#counters for precisely this reason.
In hadoop-0.20.203 it could be as big as multiple KB, but no more.
On Jun 10, 2011, at 9:49 PM, Shen LI wrote:
> Hi All,
>
> Does anyone know how many bytes does one heart beat message contain?
> And what about the return value of heart beat message?
>
> Thanks,
>
> Shen