Re: Too many open files

[Jean-Daniel Cryans <jd...@apache.org>]

Michael,

Not sure if you still have that problem, but I got it too and here is
how to fix it :
http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/

J-D

On Tue, Jan 20, 2009 at 12:26 PM, Michael Dagaev
<mi...@gmail.com> wrote:
> Hi, Stack
>
>> The 'Getting Started' for hbase advises upping file descriptors.
>
> Yes, I should have done that.
>
>> The 'files' are remote, over a TCP socket, on hdfs datanodes.
>
> I see.
>
>>> As I see, a region server holds: ~150 open epolls, ~300 open pipes,
>>> ~150 open TCP connections to itself (port 50010).
>>>
>>> Is it ok? Why does a region server need so many IPCs?
>>> Why does it use TCP connections as local IPC? Isn't it too expensive?
>>
>>
>> It has a socket per open file.  Its how hdfs works currently.
>> The local connections are probably the regionserver talking to the local datanode.
>
> Ok
>
>>> Now let's say that the region server run out of file descriptors and
>>> cannot open
>>> a new IPC. Can it continue working using ~600 IPCs it opened before?
>>>
>>>
>>
>> No.  It will fail.  Up your FDs.
>
> Anyway, it is strange that a region server needs so many FDs while we have
> only few column families. I will try to monitor FDs and gather more
> info about it.
>
> Thank you for your cooperation,
> M.
>