message transmission in Hadoop

[Da Zheng <zh...@gmail.com>]

Hello,

I monitored system calls of HDFS with systemtap and found HDFS actually sends
many 1-byte data to the network. I could also see many 8-byte and 64-byte data
written to the OS though I don't know whether they are written to the disk or
sent to the network. I did see many 8-byte data sent to the network. The number
of these data is several times more than 64KB data packet sent by HDFS.

Could anyone tell me why HDFS sends so many small packets? heartbeat messages?
RPCs? It doesn't seem to me these messages can be just 1 byte.

Thanks,
Da

Re: message transmission in Hadoop

[Da Zheng <zh...@gmail.com>]

Yes, this is exactly what I observed. reading is another problem. Thanks.

Best,
Da

On 01/30/2011 05:25 PM, Jeff Hammerbacher wrote:
> Hey Da,
>
> You may have observed https://issues.apache.org/jira/browse/HDFS-1601.
>
> Regards,
> Jeff
>
> On Fri, Jan 28, 2011 at 7:08 PM, Da Zheng<zh...@gmail.com>  wrote:
>
>> Hello,
>>
>> I monitored system calls of HDFS with systemtap and found HDFS actually
>> sends
>> many 1-byte data to the network. I could also see many 8-byte and 64-byte
>> data
>> written to the OS though I don't know whether they are written to the disk
>> or
>> sent to the network. I did see many 8-byte data sent to the network. The
>> number
>> of these data is several times more than 64KB data packet sent by HDFS.
>>
>> Could anyone tell me why HDFS sends so many small packets? heartbeat
>> messages?
>> RPCs? It doesn't seem to me these messages can be just 1 byte.
>>
>> Thanks,
>> Da
>>

Re: message transmission in Hadoop

[Da Zheng <zh...@gmail.com>]

Jeff, thanks for your reply. unfortunately, the website is in maintenance. The
reason I monitored the system calls of HDFS is to try to find out what
activities cause so much system CPU time. Other than writing to the disk and
sending and receiving packets, I cannot think of anything else that can consume
large system CPU time. Unfortunately, I wrote a simple program outputting data
(to the disk and the network) at the same rate, it can only consume far less
system CPU time.

Da


On 1/30/11 5:25 PM, Jeff Hammerbacher wrote:
> Hey Da,
> 
> You may have observed https://issues.apache.org/jira/browse/HDFS-1601.
> 
> Regards,
> Jeff
> 
> On Fri, Jan 28, 2011 at 7:08 PM, Da Zheng <zh...@gmail.com> wrote:
> 
>> Hello,
>>
>> I monitored system calls of HDFS with systemtap and found HDFS actually
>> sends
>> many 1-byte data to the network. I could also see many 8-byte and 64-byte
>> data
>> written to the OS though I don't know whether they are written to the disk
>> or
>> sent to the network. I did see many 8-byte data sent to the network. The
>> number
>> of these data is several times more than 64KB data packet sent by HDFS.
>>
>> Could anyone tell me why HDFS sends so many small packets? heartbeat
>> messages?
>> RPCs? It doesn't seem to me these messages can be just 1 byte.
>>
>> Thanks,
>> Da
>>
> 

Re: message transmission in Hadoop

[Jeff Hammerbacher <ha...@cloudera.com>]

Hey Da,

You may have observed https://issues.apache.org/jira/browse/HDFS-1601.

Regards,
Jeff

On Fri, Jan 28, 2011 at 7:08 PM, Da Zheng <zh...@gmail.com> wrote:

> Hello,
>
> I monitored system calls of HDFS with systemtap and found HDFS actually
> sends
> many 1-byte data to the network. I could also see many 8-byte and 64-byte
> data
> written to the OS though I don't know whether they are written to the disk
> or
> sent to the network. I did see many 8-byte data sent to the network. The
> number
> of these data is several times more than 64KB data packet sent by HDFS.
>
> Could anyone tell me why HDFS sends so many small packets? heartbeat
> messages?
> RPCs? It doesn't seem to me these messages can be just 1 byte.
>
> Thanks,
> Da
>