You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Marc Harris <mh...@jumptap.com> on 2008/01/28 16:11:31 UTC
What should the open file limit be for hbase
I have seen that hbase can cause "too many open file" errors. I increase
my limit to 10240 (10 times the previous limit) but still get errors.
Is there a recommended value that I should set my open files limit to?
Is there something else I can do to reduce the number of files, perhaps
with some other trade-off?
Thanks
- Marc
Re: What should the open file limit be for hbase
Posted by Marc Harris <mh...@jumptap.com>.
Well the datanode log file was full of exceptions like this:
2008-01-28 11:32:44,378 ERROR org.apache.hadoop.dfs.DataNode:
66.135.42.137:50010:DataXceiver:
java.io.FileNotFoundException: /lv_main/hadoop/dfs/data/current/subdir11/blk_-13252053410
85798084.meta (Too many open files)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:106)
at
org.apache.hadoop.dfs.FSDataset.getMetaDataInputStream(FSDataset.java:481)
at org.apache.hadoop.dfs.DataNode
$BlockSender.<init>(DataNode.java:1298)
at org.apache.hadoop.dfs.DataNode
$DataXceiver.readBlock(DataNode.java:913)
at org.apache.hadoop.dfs.DataNode
$DataXceiver.run(DataNode.java:865)
at java.lang.Thread.run(Thread.java:595)
2008-01-28 11:32:45,060 ERROR org.apache.hadoop.dfs.DataNode:
66.135.42.137:50010:DataXceiver: java.io.IOException: Too many open
files
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:850)
at org.apache.hadoop.dfs.FSDataset
$FSVolume.createTmpFile(FSDataset.java:329)
at
org.apache.hadoop.dfs.FSDataset.createTmpFile(FSDataset.java:637)
at
org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:613)
at org.apache.hadoop.dfs.DataNode
$BlockReceiver.<init>(DataNode.java:1498)
at org.apache.hadoop.dfs.DataNode
$DataXceiver.writeBlock(DataNode.java:973)
at org.apache.hadoop.dfs.DataNode
$DataXceiver.run(DataNode.java:868)
at java.lang.Thread.run(Thread.java:595)
2008-01-28 11:32:45,466 ERROR org.apache.hadoop.dfs.DataNode:
66.135.42.137:50010:DataXceiver:
java.io.FileNotFoundException: /lv_main/hadoop/dfs/data/tmp/blk_-1866322536983816592.meta
(Too many open files)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
at
org.apache.hadoop.dfs.FSDataset.createBlockWriteStreams(FSDataset.java:569)
at
org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:624)
at org.apache.hadoop.dfs.DataNode
$BlockReceiver.<init>(DataNode.java:1498)
at org.apache.hadoop.dfs.DataNode
$DataXceiver.writeBlock(DataNode.java:973)
at org.apache.hadoop.dfs.DataNode
$DataXceiver.run(DataNode.java:868)
at java.lang.Thread.run(Thread.java:595)
Stupidly I did not keep a record of the lsof output. All I remember is
that most of the open files seemed to be sockets not local files. I can
send you log files if you want them, but not the lsof output.
- Marc
On Mon, 2008-01-28 at 10:06 -0800, stack wrote:
> Exceptions are complaints about too many open files? Looks like you
> have to up your ulimit given the numbers you are showing below. Odd
> though is that its the datanode that is overflowing the limit. Want to
> send me the lsof offlist so I can take a gander?
>
> Thanks,
> St.Ack
>
>
> Marc Harris wrote:
> > Sorry, I should have said 4 families in one table, obviously, not 4
> > regions in one table.
> > - Marc
> >
> >
> > On Mon, 2008-01-28 at 11:57 -0500, Marc Harris wrote:
> >
> >
> >> My schema is very simple: 4 regions in one table.
> >>
> >> create table pagefetch (
> >> info MAX_VERSIONS=1,
> >> data MAX_VERSIONS=1,
> >> headers MAX_VERSIONS=1,
> >> redirects MAX_VERSIONS=1
> >> );
> >>
> >> I am running hadoop in distributed configuration, but with only one data
> >> node.
> >> I am running hbase with two region servers (one of which is one the same
> >> machine as hadoop).
> >> I am seeing the exceptions in my datanode log file, by the way, not my
> >> regionserver log file
> >> "lsof -p REGIONSERVER_PID | wc -l" gave 479
> >> "lsof -p DATANODE_PID | wc -l" gave 10287
> >>
> >> - Marc
> >>
> >>
> >>
> >>
> >> On Mon, 2008-01-28 at 08:13 -0800, stack wrote:
> >>
> >>
> >>> Hey Marc:
> >>>
> >>> You are still seeing 'too many open files'? Whats your schema look
> >>> like. I added to http://wiki.apache.org/hadoop/Hbase/FAQ#5 a rough
> >>> formula for counting how many open mapfiles in a running regionserver.
> >>>
> >>> Currently, your only recourse is upping the ulimit. Addressing this
> >>> scaling barrier will be a focus of next hbase release.
> >>>
> >>> St.Ack
> >>>
> >>>
> >>>
> >>> Marc Harris wrote:
> >>>
> >>>> I have seen that hbase can cause "too many open file" errors. I increase
> >>>> my limit to 10240 (10 times the previous limit) but still get errors.
> >>>>
> >>>> Is there a recommended value that I should set my open files limit to?
> >>>> Is there something else I can do to reduce the number of files, perhaps
> >>>> with some other trade-off?
> >>>>
> >>>> Thanks
> >>>> - Marc
> >>>>
> >>>>
> >>>>
> >>>>
> >
> >
>
Re: What should the open file limit be for hbase
Posted by stack <st...@duboce.net>.
Exceptions are complaints about too many open files? Looks like you
have to up your ulimit given the numbers you are showing below. Odd
though is that its the datanode that is overflowing the limit. Want to
send me the lsof offlist so I can take a gander?
Thanks,
St.Ack
Marc Harris wrote:
> Sorry, I should have said 4 families in one table, obviously, not 4
> regions in one table.
> - Marc
>
>
> On Mon, 2008-01-28 at 11:57 -0500, Marc Harris wrote:
>
>
>> My schema is very simple: 4 regions in one table.
>>
>> create table pagefetch (
>> info MAX_VERSIONS=1,
>> data MAX_VERSIONS=1,
>> headers MAX_VERSIONS=1,
>> redirects MAX_VERSIONS=1
>> );
>>
>> I am running hadoop in distributed configuration, but with only one data
>> node.
>> I am running hbase with two region servers (one of which is one the same
>> machine as hadoop).
>> I am seeing the exceptions in my datanode log file, by the way, not my
>> regionserver log file
>> "lsof -p REGIONSERVER_PID | wc -l" gave 479
>> "lsof -p DATANODE_PID | wc -l" gave 10287
>>
>> - Marc
>>
>>
>>
>>
>> On Mon, 2008-01-28 at 08:13 -0800, stack wrote:
>>
>>
>>> Hey Marc:
>>>
>>> You are still seeing 'too many open files'? Whats your schema look
>>> like. I added to http://wiki.apache.org/hadoop/Hbase/FAQ#5 a rough
>>> formula for counting how many open mapfiles in a running regionserver.
>>>
>>> Currently, your only recourse is upping the ulimit. Addressing this
>>> scaling barrier will be a focus of next hbase release.
>>>
>>> St.Ack
>>>
>>>
>>>
>>> Marc Harris wrote:
>>>
>>>> I have seen that hbase can cause "too many open file" errors. I increase
>>>> my limit to 10240 (10 times the previous limit) but still get errors.
>>>>
>>>> Is there a recommended value that I should set my open files limit to?
>>>> Is there something else I can do to reduce the number of files, perhaps
>>>> with some other trade-off?
>>>>
>>>> Thanks
>>>> - Marc
>>>>
>>>>
>>>>
>>>>
>
>
Re: What should the open file limit be for hbase
Posted by Marc Harris <mh...@jumptap.com>.
Sorry, I should have said 4 families in one table, obviously, not 4
regions in one table.
- Marc
On Mon, 2008-01-28 at 11:57 -0500, Marc Harris wrote:
> My schema is very simple: 4 regions in one table.
>
> create table pagefetch (
> info MAX_VERSIONS=1,
> data MAX_VERSIONS=1,
> headers MAX_VERSIONS=1,
> redirects MAX_VERSIONS=1
> );
>
> I am running hadoop in distributed configuration, but with only one data
> node.
> I am running hbase with two region servers (one of which is one the same
> machine as hadoop).
> I am seeing the exceptions in my datanode log file, by the way, not my
> regionserver log file
> "lsof -p REGIONSERVER_PID | wc -l" gave 479
> "lsof -p DATANODE_PID | wc -l" gave 10287
>
> - Marc
>
>
>
>
> On Mon, 2008-01-28 at 08:13 -0800, stack wrote:
>
> > Hey Marc:
> >
> > You are still seeing 'too many open files'? Whats your schema look
> > like. I added to http://wiki.apache.org/hadoop/Hbase/FAQ#5 a rough
> > formula for counting how many open mapfiles in a running regionserver.
> >
> > Currently, your only recourse is upping the ulimit. Addressing this
> > scaling barrier will be a focus of next hbase release.
> >
> > St.Ack
> >
> >
> >
> > Marc Harris wrote:
> > > I have seen that hbase can cause "too many open file" errors. I increase
> > > my limit to 10240 (10 times the previous limit) but still get errors.
> > >
> > > Is there a recommended value that I should set my open files limit to?
> > > Is there something else I can do to reduce the number of files, perhaps
> > > with some other trade-off?
> > >
> > > Thanks
> > > - Marc
> > >
> > >
> > >
> >
Re: What should the open file limit be for hbase
Posted by Marc Harris <mh...@jumptap.com>.
My schema is very simple: 4 regions in one table.
create table pagefetch (
info MAX_VERSIONS=1,
data MAX_VERSIONS=1,
headers MAX_VERSIONS=1,
redirects MAX_VERSIONS=1
);
I am running hadoop in distributed configuration, but with only one data
node.
I am running hbase with two region servers (one of which is one the same
machine as hadoop).
I am seeing the exceptions in my datanode log file, by the way, not my
regionserver log file
"lsof -p REGIONSERVER_PID | wc -l" gave 479
"lsof -p DATANODE_PID | wc -l" gave 10287
- Marc
On Mon, 2008-01-28 at 08:13 -0800, stack wrote:
> Hey Marc:
>
> You are still seeing 'too many open files'? Whats your schema look
> like. I added to http://wiki.apache.org/hadoop/Hbase/FAQ#5 a rough
> formula for counting how many open mapfiles in a running regionserver.
>
> Currently, your only recourse is upping the ulimit. Addressing this
> scaling barrier will be a focus of next hbase release.
>
> St.Ack
>
>
>
> Marc Harris wrote:
> > I have seen that hbase can cause "too many open file" errors. I increase
> > my limit to 10240 (10 times the previous limit) but still get errors.
> >
> > Is there a recommended value that I should set my open files limit to?
> > Is there something else I can do to reduce the number of files, perhaps
> > with some other trade-off?
> >
> > Thanks
> > - Marc
> >
> >
> >
>
Re: What should the open file limit be for hbase
Posted by stack <st...@duboce.net>.
Hey Marc:
You are still seeing 'too many open files'? Whats your schema look
like. I added to http://wiki.apache.org/hadoop/Hbase/FAQ#5 a rough
formula for counting how many open mapfiles in a running regionserver.
Currently, your only recourse is upping the ulimit. Addressing this
scaling barrier will be a focus of next hbase release.
St.Ack
Marc Harris wrote:
> I have seen that hbase can cause "too many open file" errors. I increase
> my limit to 10240 (10 times the previous limit) but still get errors.
>
> Is there a recommended value that I should set my open files limit to?
> Is there something else I can do to reduce the number of files, perhaps
> with some other trade-off?
>
> Thanks
> - Marc
>
>
>