socket buffer sizes hardcoded

[Jay Kreps <ja...@gmail.com>]

Hey Guys,

I am interested in increasing the throughput of an HDFS read while
transferring data between datacenters that are geographically far
apart and hence have a network latency of around 60ms. I see in the
HDFS code that the DFSClient and DataNode seem to hardcode their
socket buffer sizes to 128KB (DFSClient.createBlockOutputStream and
DataNode.startDataNode). Is there a reason for this?

I want to expose this value as a configurable property so that when i
read over the high-latency link I can set the ideal buffer size for
this particular application (around 800KB for our desired bandwidth).
Is there a reason this is not done currently? Would you take a patch
that added this property? Am I looking at totally the wrong code?

-Jay

Re: socket buffer sizes hardcoded

[Dhruba Borthakur <dh...@gmail.com>]

Making it configurable seems like a good thing. There is a JIRA (owned by
Sanjay) that describes that some of these configuration variables on the
client side might become "undocumented"; tjois means that they might change
semantics from a release to another.

thanks,
dhruba


On Wed, Sep 2, 2009 at 7:45 PM, Jay Kreps <ja...@gmail.com> wrote:

> Hey Guys,
>
> I am interested in increasing the throughput of an HDFS read while
> transferring data between datacenters that are geographically far
> apart and hence have a network latency of around 60ms. I see in the
> HDFS code that the DFSClient and DataNode seem to hardcode their
> socket buffer sizes to 128KB (DFSClient.createBlockOutputStream and
> DataNode.startDataNode). Is there a reason for this?
>
> I want to expose this value as a configurable property so that when i
> read over the high-latency link I can set the ideal buffer size for
> this particular application (around 800KB for our desired bandwidth).
> Is there a reason this is not done currently? Would you take a patch
> that added this property? Am I looking at totally the wrong code?
>
> -Jay
>