You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jean-Adrien <ad...@jeanjean.ch> on 2009/01/09 14:24:07 UTC
Datanode Xceivers
Hi everybody,
I saw that you put some advises concerning the Hadoop settings when one has
a problem of max xceivers reached, in the troubleshooting section of the
wiki.
About this topic, I recently post a question in hadoop-core user mailing
list about their 'xcievers' thread behavior, since I still had to increase
their amount as my HBase table grows, in order to avoid to reach the limit
at startup time. And therefore my jvm use a lot of virtual memory (actually
with 500MB for the heap, 1100 threads allocate 2GB virtual memory). This
evenutally yields to swap and failure.
Here is the link to my post. With a graph showing the number of thread the
datanode creates when I start hbase.
http://www.nabble.com/xceiverCount-limit-reason-td21349807.html#a21352818
You can see that all threads are created at HBase startup time, and, if the
timeout ( dfs.datanode.socket.write.timeout
) is set, they all ends with a timeout failure.
The question for HBase is, why are the connection with hadoop kept open (and
the thread as well) ? Does it happen only in my case ?
I think that Slava has the same problem. But I don't think everybody does,
since the cluster could not run without disabling the timeout parameter
dfs.datanode.socket.write.timeout
Anybody made those observations ?
Thanks
Jean-Adrien
--
View this message in context: http://www.nabble.com/Datanode-Xceivers-tp21372227p21372227.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: Datanode Xceivers
Posted by Jean-Adrien <ad...@jeanjean.ch>.
Hello all.
Thanks a lot for your tests and information. I understand better now the
reason of that.
I'll turn to HBase/Hadoop 0.19.0 as soon as possible. In the meantime it is
ok for me to keep the timeout *dfs.datanode.socket.write.timeout* disabled.
As soon I upgrade to 0.19.0, I'll enable to timeout, and see if it fixes the
problem of "Premeture EOF".
Have a nice day.
-- J.-A.
stack-3 wrote:
>
> Jean-Adrien:
>
> For kicks, I tried your test in hadoop 0.19.0 (and hbase trunk). Using
> default for *dfs.datanode.socket.write.timeout*, 8 minutes, Scenario B in
> your message to hadoop list, I started up a loaded cluster. After all had
> settled I counted DataXceiver threads in datanode. Each of my datanodes
> had
> more than 100 instances. I let it sit > 8 minutes. Now the datanodes had
> but one or two DataXceiver threads (I could see all the ERROR timeouts
> tripping in the datanode log). I started up a scan of all content in the
> table. It ran without issue. No exceptions in the regionserver logs.
> Number of DataXceiver threads came and went over life of the scan.
>
> So, there is still the big datanode memory/thread pressure on startup and
> then there are the issues where there will be extra latency reestablishing
> timedout readers.
>
> Scenario C where you shutdown hbase, for me, also shuts down all resources
> in datanode.
>
> St.Ack
>
>
>
> On Sun, Jan 11, 2009 at 11:34 PM, stack <st...@duboce.net> wrote:
>
>> Luo Ning, over the weekend, has made some comments you might be
>> interested
>> in over in HBASE-24 Jean-Adrien.
>> St.Ack
>>
>>
>>
>> Jean-Adrien wrote:
>>
>>> Hi everybody,
>>>
>>> I saw that you put some advises concerning the Hadoop settings when one
>>> has
>>> a problem of max xceivers reached, in the troubleshooting section of the
>>> wiki.
>>>
>>> About this topic, I recently post a question in hadoop-core user mailing
>>> list about their 'xcievers' thread behavior, since I still had to
>>> increase
>>> their amount as my HBase table grows, in order to avoid to reach the
>>> limit
>>> at startup time. And therefore my jvm use a lot of virtual memory
>>> (actually
>>> with 500MB for the heap, 1100 threads allocate 2GB virtual memory). This
>>> evenutally yields to swap and failure.
>>>
>>> Here is the link to my post. With a graph showing the number of thread
>>> the
>>> datanode creates when I start hbase.
>>> http://www.nabble.com/xceiverCount-limit-reason-td21349807.html#a21352818
>>>
>>> You can see that all threads are created at HBase startup time, and, if
>>> the
>>> timeout ( dfs.datanode.socket.write.timeout
>>> ) is set, they all ends with a timeout failure.
>>>
>>> The question for HBase is, why are the connection with hadoop kept open
>>> (and
>>> the thread as well) ? Does it happen only in my case ?
>>> I think that Slava has the same problem. But I don't think everybody
>>> does,
>>> since the cluster could not run without disabling the timeout parameter
>>> dfs.datanode.socket.write.timeout
>>>
>>> Anybody made those observations ?
>>> Thanks
>>>
>>> Jean-Adrien
>>>
>>>
>>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Datanode-Xceivers-tp21372227p21567289.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: Datanode Xceivers
Posted by stack <st...@duboce.net>.
Jean-Adrien:
For kicks, I tried your test in hadoop 0.19.0 (and hbase trunk). Using
default for *dfs.datanode.socket.write.timeout*, 8 minutes, Scenario B in
your message to hadoop list, I started up a loaded cluster. After all had
settled I counted DataXceiver threads in datanode. Each of my datanodes had
more than 100 instances. I let it sit > 8 minutes. Now the datanodes had
but one or two DataXceiver threads (I could see all the ERROR timeouts
tripping in the datanode log). I started up a scan of all content in the
table. It ran without issue. No exceptions in the regionserver logs.
Number of DataXceiver threads came and went over life of the scan.
So, there is still the big datanode memory/thread pressure on startup and
then there are the issues where there will be extra latency reestablishing
timedout readers.
Scenario C where you shutdown hbase, for me, also shuts down all resources
in datanode.
St.Ack
On Sun, Jan 11, 2009 at 11:34 PM, stack <st...@duboce.net> wrote:
> Luo Ning, over the weekend, has made some comments you might be interested
> in over in HBASE-24 Jean-Adrien.
> St.Ack
>
>
>
> Jean-Adrien wrote:
>
>> Hi everybody,
>>
>> I saw that you put some advises concerning the Hadoop settings when one
>> has
>> a problem of max xceivers reached, in the troubleshooting section of the
>> wiki.
>>
>> About this topic, I recently post a question in hadoop-core user mailing
>> list about their 'xcievers' thread behavior, since I still had to increase
>> their amount as my HBase table grows, in order to avoid to reach the limit
>> at startup time. And therefore my jvm use a lot of virtual memory
>> (actually
>> with 500MB for the heap, 1100 threads allocate 2GB virtual memory). This
>> evenutally yields to swap and failure.
>>
>> Here is the link to my post. With a graph showing the number of thread the
>> datanode creates when I start hbase.
>> http://www.nabble.com/xceiverCount-limit-reason-td21349807.html#a21352818
>>
>> You can see that all threads are created at HBase startup time, and, if
>> the
>> timeout ( dfs.datanode.socket.write.timeout
>> ) is set, they all ends with a timeout failure.
>>
>> The question for HBase is, why are the connection with hadoop kept open
>> (and
>> the thread as well) ? Does it happen only in my case ?
>> I think that Slava has the same problem. But I don't think everybody does,
>> since the cluster could not run without disabling the timeout parameter
>> dfs.datanode.socket.write.timeout
>>
>> Anybody made those observations ?
>> Thanks
>>
>> Jean-Adrien
>>
>>
>>
>
>
Re: Datanode Xceivers
Posted by stack <st...@duboce.net>.
Luo Ning, over the weekend, has made some comments you might be
interested in over in HBASE-24 Jean-Adrien.
St.Ack
Jean-Adrien wrote:
> Hi everybody,
>
> I saw that you put some advises concerning the Hadoop settings when one has
> a problem of max xceivers reached, in the troubleshooting section of the
> wiki.
>
> About this topic, I recently post a question in hadoop-core user mailing
> list about their 'xcievers' thread behavior, since I still had to increase
> their amount as my HBase table grows, in order to avoid to reach the limit
> at startup time. And therefore my jvm use a lot of virtual memory (actually
> with 500MB for the heap, 1100 threads allocate 2GB virtual memory). This
> evenutally yields to swap and failure.
>
> Here is the link to my post. With a graph showing the number of thread the
> datanode creates when I start hbase.
> http://www.nabble.com/xceiverCount-limit-reason-td21349807.html#a21352818
>
> You can see that all threads are created at HBase startup time, and, if the
> timeout ( dfs.datanode.socket.write.timeout
> ) is set, they all ends with a timeout failure.
>
> The question for HBase is, why are the connection with hadoop kept open (and
> the thread as well) ? Does it happen only in my case ?
> I think that Slava has the same problem. But I don't think everybody does,
> since the cluster could not run without disabling the timeout parameter
> dfs.datanode.socket.write.timeout
>
> Anybody made those observations ?
> Thanks
>
> Jean-Adrien
>
>
Re: Datanode Xceivers
Posted by stack <st...@duboce.net>.
Raghu Angadi wrote:
>> I haven't been paying attention of late. Thanks for bringing it up
>> Jean-Adrien. Lets try and figure it (I 'thought' that the timer over
>> on the datanode would close idle sockets but that subsequent accesses
>> would revive the connection but that doesn't seem to be the case
>> going by your hadoop posting).
> For correct behavior with multiple timeouts, you need
> https://issues.apache.org/jira/browse/HADOOP-3831 . I don't know why
> that didn't go into 0.18.
>
> Raghu.
Let me redo Jean-Adrien's experiement with hadoop 0.19.0 (with
HADOOP-3831) in place (Thanks for coming by Raghu).
St.Ack
Re: Datanode Xceivers
Posted by Raghu Angadi <ra...@yahoo-inc.com>.
stack wrote:
> Jean-Adrien wrote:
>> lava has the same problem. But I don't think everybody does,
>> since the cluster could not run without disabling the timeout parameter
>> dfs.datanode.socket.write.timeout
>>
>> Anybody made those observations ?
>>
>
> I haven't been paying attention of late. Thanks for bringing it up
> Jean-Adrien. Lets try and figure it (I 'thought' that the timer over on
> the datanode would close idle sockets but that subsequent accesses would
> revive the connection but that doesn't seem to be the case going by your
> hadoop posting).
For correct behavior with multiple timeouts, you need
https://issues.apache.org/jira/browse/HADOOP-3831 . I don't know why
that didn't go into 0.18.
Raghu.
> St.Ack
Re: Datanode Xceivers
Posted by stack <st...@duboce.net>.
Jean-Adrien wrote:
> Hi everybody,
>
> I saw that you put some advises concerning the Hadoop settings when one has
> a problem of max xceivers reached, in the troubleshooting section of the
> wiki.
>
Yes. Thanks to your research Jean-Adrien. And have you seen the
addition made by Andrew Purtell suggesting upping the datanode listeners?
> About this topic, I recently post a question in hadoop-core user mailing
> list about their 'xcievers' thread behavior, since I still had to increase
> their amount as my HBase table grows, in order to avoid to reach the limit
> at startup time. And therefore my jvm use a lot of virtual memory (actually
> with 500MB for the heap, 1100 threads allocate 2GB virtual memory). This
> evenutally yields to swap and failure.
>
>
Yeah. That makes sense (Have you tried setting thread stack size down
-- -Xss -- so less outside-of-the-heap memory is used?)
> Here is the link to my post. With a graph showing the number of thread the
> datanode creates when I start hbase.
> http://www.nabble.com/xceiverCount-limit-reason-td21349807.html#a21352818
>
> You can see that all threads are created at HBase startup time, and, if the
> timeout ( dfs.datanode.socket.write.timeout
> ) is set, they all ends with a timeout failure.
>
> The question for HBase is, why are the connection with hadoop kept open (and
> the thread as well) ? Does it happen only in my case ?
>
No. Happens for everyone. HBase keeps open its connection to every
StoreFile. We do this to avoid paying the open cost every time a file
is accessed primarily to improve random-access performance. StoreFiles
in hbase are based on Hadoop MapFile. MapFile is two SequenceFiles --
data and index. An open would require at least a trip to namenode per
SequenceFile to learn blocks that make up a file, then trip to the
holding datanodes first to read in index if a random-access and then to
the target block once its location was found. Instead, per StoreFile,
on open, we read in the index (and then close the index file) and then
keep up the DFSClient connection to the datafile so block locations are
kept over in the hbase regionserver.
Keeping open a permanent connection to the store file costs us. Users
will trip over the 'too many open files...' pretty early on unless they
up their ulimit for file descriptors. Also, keeping the index in memory
as we currently do is the main cause of heap usage -- particularly if
cells are small. Then there is the cost over in HDFS which is what you
are bringing up here.
> lava has the same problem. But I don't think everybody does,
> since the cluster could not run without disabling the timeout parameter
> dfs.datanode.socket.write.timeout
>
> Anybody made those observations ?
>
I haven't been paying attention of late. Thanks for bringing it up
Jean-Adrien. Lets try and figure it (I 'thought' that the timer over on
the datanode would close idle sockets but that subsequent accesses would
revive the connection but that doesn't seem to be the case going by your
hadoop posting).
St.Ack