Lots of connections from datanodes in TIME_WAIT

[da...@internode.on.net]

Hi all,
We have a 10 node cluster and I noticed that there are a large number of connections in TIME_WAIT status,  over 400 to more precise.  Does 
anyone
else see so many unclosed connections or have I done something wrong in my config.  It appears to be coming from and going to other 
datanodes.
I was running the javasort benchmark when I noticed it.  Any advice appreciated.

Regards
Damien

Re: Lots of connections from datanodes in TIME_WAIT

[Ted Dunning <te...@gmail.com>]

When it needs a socket and all are in TIME_WAIT, it basically switches to
FIFO strategy instead of waiting forever.

This is critical when using Linux as a load balancer on large web
applications.  You can easily have 100,000 unique connections in a few
minutes.

On Tue, Sep 16, 2008 at 6:09 PM, Raghu Angadi <ra...@yahoo-inc.com> wrote:

> Ted Dunning wrote:
>
>> There are OS settings, but modern Linux kernels automatically adjust the
>> setting if necessary.
>>
>
> That is good. Do you know what it does?
>
> When I was replying to the original mail thought Linux might actively
> remove connections from this state if the number of such connections reaches
> a limit.. otherwise it could be a pretty bad scalability limit for some
> applications.
>
> Raghu.
>
>
>  On Tue, Sep 16, 2008 at 8:38 AM, Raghu Angadi <ra...@yahoo-inc.com>
>> wrote:
>>
>>  Thats ok. Its a normal TCP state for the closed connections. There is
>>> probably some OS level setting to reduce the amount of time these
>>> connections stay in this state.
>>>
>>> It does not indicate a bug in application.
>>>
>>> Raghu.
>>>
>>>
>>> damien.cooke@internode.on.net wrote:
>>>
>>>  Hi all,
>>>> We have a 10 node cluster and I noticed that there are a large number of
>>>> connections in TIME_WAIT status,  over 400 to more precise.  Does anyone
>>>> else see so many unclosed connections or have I done something wrong in
>>>> my
>>>> config.  It appears to be coming from and going to other datanodes.
>>>> I was running the javasort benchmark when I noticed it.  Any advice
>>>> appreciated.
>>>>
>>>> Regards
>>>> Damien
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>
>>
>


-- 
ted

Re: Lots of connections from datanodes in TIME_WAIT

[Raghu Angadi <ra...@yahoo-inc.com>]

Ted Dunning wrote:
> There are OS settings, but modern Linux kernels automatically adjust the
> setting if necessary.

That is good. Do you know what it does?

When I was replying to the original mail thought Linux might actively 
remove connections from this state if the number of such connections 
reaches a limit.. otherwise it could be a pretty bad scalability limit 
for some applications.

Raghu.

> On Tue, Sep 16, 2008 at 8:38 AM, Raghu Angadi <ra...@yahoo-inc.com> wrote:
> 
>> Thats ok. Its a normal TCP state for the closed connections. There is
>> probably some OS level setting to reduce the amount of time these
>> connections stay in this state.
>>
>> It does not indicate a bug in application.
>>
>> Raghu.
>>
>>
>> damien.cooke@internode.on.net wrote:
>>
>>> Hi all,
>>> We have a 10 node cluster and I noticed that there are a large number of
>>> connections in TIME_WAIT status,  over 400 to more precise.  Does anyone
>>> else see so many unclosed connections or have I done something wrong in my
>>> config.  It appears to be coming from and going to other datanodes.
>>> I was running the javasort benchmark when I noticed it.  Any advice
>>> appreciated.
>>>
>>> Regards
>>> Damien
>>>
>>>
>>>
>>>
>>>
> 
> 

Re: Lots of connections from datanodes in TIME_WAIT

[Ted Dunning <te...@gmail.com>]

There are OS settings, but modern Linux kernels automatically adjust the
setting if necessary.

On Tue, Sep 16, 2008 at 8:38 AM, Raghu Angadi <ra...@yahoo-inc.com> wrote:

>
> Thats ok. Its a normal TCP state for the closed connections. There is
> probably some OS level setting to reduce the amount of time these
> connections stay in this state.
>
> It does not indicate a bug in application.
>
> Raghu.
>
>
> damien.cooke@internode.on.net wrote:
>
>> Hi all,
>> We have a 10 node cluster and I noticed that there are a large number of
>> connections in TIME_WAIT status,  over 400 to more precise.  Does anyone
>> else see so many unclosed connections or have I done something wrong in my
>> config.  It appears to be coming from and going to other datanodes.
>> I was running the javasort benchmark when I noticed it.  Any advice
>> appreciated.
>>
>> Regards
>> Damien
>>
>>
>>
>>
>>
>


-- 
ted

Re: Lots of connections from datanodes in TIME_WAIT

[Raghu Angadi <ra...@yahoo-inc.com>]

Thats ok. Its a normal TCP state for the closed connections. There is 
probably some OS level setting to reduce the amount of time these 
connections stay in this state.

It does not indicate a bug in application.

Raghu.

damien.cooke@internode.on.net wrote:
> Hi all,
> We have a 10 node cluster and I noticed that there are a large number of connections in TIME_WAIT status,  over 400 to more precise.  Does 
> anyone
> else see so many unclosed connections or have I done something wrong in my config.  It appears to be coming from and going to other 
> datanodes.
> I was running the javasort benchmark when I noticed it.  Any advice appreciated.
> 
> Regards
> Damien
> 
> 
> 
>