You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by jason lu <lj...@gmail.com> on 2015/05/20 05:30:12 UTC
“dfs.namenode.service.handler.count” and “dfs.namenode.handler.count”
Hi,
what is the difference between “dfs.namenode.service.handler.count” and “dfs.namenode.handler.count” in hdfs-site.xml?
I found this in stackoverflow:
The RPC Server needs threads to handle requests- Hadoop ships with his own RPC framework and you can configure this with the dfs.namenode.service.handler.count for the datanodes it is default set to 10. Or you can configure this dfs.namenode.handler.count for other clients, like MapReduce jobs, JobClients that want to run a job. When a request comes in and it want to create a new handler, it go may out of memory (new Threads are also allocating a good chunk of stack space, maybe you need to increase this).
I can’t find the dfs.namenode.service.handler.count in hdfs-site.xml of hadoop 2.3, is is deprecated? but I found it in the source.
BTW, any suggestion for these parameters?
thanks.
Re: “dfs.namenode.service.handler.count” and “dfs.namenode.handler.count”
Posted by Akira AJISAKA <aj...@oss.nttdata.co.jp>.
RPC Response from NameNode can become slow.
Timeouts can happen in the worst case.
Regards,
Akira
On 5/20/15 18:14, jason lu wrote:
> thanks ajisakaa,
>
> what will happen if the number of “dfs.namenode.handler.count” is too
> low to allocate channel to client?
>
> thanks.
>
>
>> 在 2015年5月20日，15:10，ajisakaa <ajisakaa@oss.nttdata.co.jp
>> <ma...@oss.nttdata.co.jp>> 写道：
>>
>> dfs.namenode.handler.count
>
Re: “dfs.namenode.service.handler.count” and “dfs.namenode.handler.count”
Posted by Akira AJISAKA <aj...@oss.nttdata.co.jp>.
RPC Response from NameNode can become slow.
Timeouts can happen in the worst case.
Regards,
Akira
On 5/20/15 18:14, jason lu wrote:
> thanks ajisakaa,
>
> what will happen if the number of “dfs.namenode.handler.count” is too
> low to allocate channel to client?
>
> thanks.
>
>
>> 在 2015年5月20日，15:10，ajisakaa <ajisakaa@oss.nttdata.co.jp
>> <ma...@oss.nttdata.co.jp>> 写道：
>>
>> dfs.namenode.handler.count
>
Re: “dfs.namenode.service.handler.count” and “dfs.namenode.handler.count”
Posted by Akira AJISAKA <aj...@oss.nttdata.co.jp>.
RPC Response from NameNode can become slow.
Timeouts can happen in the worst case.
Regards,
Akira
On 5/20/15 18:14, jason lu wrote:
> thanks ajisakaa,
>
> what will happen if the number of “dfs.namenode.handler.count” is too
> low to allocate channel to client?
>
> thanks.
>
>
>> 在 2015年5月20日，15:10，ajisakaa <ajisakaa@oss.nttdata.co.jp
>> <ma...@oss.nttdata.co.jp>> 写道：
>>
>> dfs.namenode.handler.count
>
Re: “dfs.namenode.service.handler.count” and “dfs.namenode.handler.count”
Posted by Akira AJISAKA <aj...@oss.nttdata.co.jp>.
RPC Response from NameNode can become slow.
Timeouts can happen in the worst case.
Regards,
Akira
On 5/20/15 18:14, jason lu wrote:
> thanks ajisakaa,
>
> what will happen if the number of “dfs.namenode.handler.count” is too
> low to allocate channel to client?
>
> thanks.
>
>
>> 在 2015年5月20日，15:10，ajisakaa <ajisakaa@oss.nttdata.co.jp
>> <ma...@oss.nttdata.co.jp>> 写道：
>>
>> dfs.namenode.handler.count
>
Re: “dfs.namenode.service.handler.count” and “dfs.namenode.handler.count”
Posted by jason lu <lj...@gmail.com>.
thanks ajisakaa,
what will happen if the number of “dfs.namenode.handler.count” is too low to allocate channel to client?
thanks.
> 在 2015年5月20日，15:10，ajisakaa <aj...@oss.nttdata.co.jp> 写道：
>
> dfs.namenode.handler.count
Re: “dfs.namenode.service.handler.count” and “dfs.namenode.handler.count”
Posted by jason lu <lj...@gmail.com>.
thanks ajisakaa,
what will happen if the number of “dfs.namenode.handler.count” is too low to allocate channel to client?
thanks.
> 在 2015年5月20日，15:10，ajisakaa <aj...@oss.nttdata.co.jp> 写道：
>
> dfs.namenode.handler.count
Re: “dfs.namenode.service.handler.count” and “dfs.namenode.handler.count”
Posted by jason lu <lj...@gmail.com>.
thanks ajisakaa,
what will happen if the number of “dfs.namenode.handler.count” is too low to allocate channel to client?
thanks.
> 在 2015年5月20日，15:10，ajisakaa <aj...@oss.nttdata.co.jp> 写道：
>
> dfs.namenode.handler.count
Re: “dfs.namenode.service.handler.count” and “dfs.namenode.handler.count”
Posted by jason lu <lj...@gmail.com>.
thanks ajisakaa,
what will happen if the number of “dfs.namenode.handler.count” is too low to allocate channel to client?
thanks.
> 在 2015年5月20日，15:10，ajisakaa <aj...@oss.nttdata.co.jp> 写道：
>
> dfs.namenode.handler.count
Re: “dfs.namenode.service.handler.count” and “dfs.namenode.handler.count”
Posted by ajisakaa <aj...@oss.nttdata.co.jp>.
Hi jason,
Looked around the source-code, both parameters are not deprecated.
dfs.namenode.service.handler.count : Specifies the number of
threads for NameNode to handle RPC requests from DataNodes,
standby NameNode, and all other non-client nodes
(BackupNode and SecondaryNameNode). Default to 10.
This parameter is valid only if dfs.namenode.servicerpc-address
is configured. If the parameter is configured, NameNode
initializes extra RPC server to handle requests from
non-client nodes.
dfs.namenode.handler.count : Specifies the number of threads
for NameNode to handle RPC requests. Default to 10.
I'll file a jira to document dfs.namenode.service.handler.count.
Regards,
Akira
On 05/20/2015 12:30 PM, jason lu wrote:
> Hi,
>
> what is the difference between “dfs.namenode.service.handler.count”
> and “dfs.namenode.handler.count” in hdfs-site.xml?
> I found this in stackoverflow:
> # The RPC Server needs threads to handle requests- Hadoop ships with his
> own RPC framework and you can configure this with the
> |dfs.namenode.service.handler.count| *for the datanodes* it is default
> set to 10. Or you can configure this |dfs.namenode.handler.count| *for
> other clients*, like MapReduce jobs, |JobClients| that want to run a
> job. When a request comes in and it want to create a new handler, it
> go may out of memory (new Threads are also allocating a good chunk of
> stack space, maybe you need to increase this).
>
> I can’t find the dfs.namenode.service.handler.count in hdfs-site.xml
> of hadoop 2.3, is is deprecated? but I found it in the source.
>
> BTW, any suggestion for these parameters?
>
> thanks.
>
>
Re: “dfs.namenode.service.handler.count” and “dfs.namenode.handler.count”
Posted by ajisakaa <aj...@oss.nttdata.co.jp>.
Hi jason,
Looked around the source-code, both parameters are not deprecated.
dfs.namenode.service.handler.count : Specifies the number of
threads for NameNode to handle RPC requests from DataNodes,
standby NameNode, and all other non-client nodes
(BackupNode and SecondaryNameNode). Default to 10.
This parameter is valid only if dfs.namenode.servicerpc-address
is configured. If the parameter is configured, NameNode
initializes extra RPC server to handle requests from
non-client nodes.
dfs.namenode.handler.count : Specifies the number of threads
for NameNode to handle RPC requests. Default to 10.
I'll file a jira to document dfs.namenode.service.handler.count.
Regards,
Akira
On 05/20/2015 12:30 PM, jason lu wrote:
> Hi,
>
> what is the difference between “dfs.namenode.service.handler.count”
> and “dfs.namenode.handler.count” in hdfs-site.xml?
> I found this in stackoverflow:
> # The RPC Server needs threads to handle requests- Hadoop ships with his
> own RPC framework and you can configure this with the
> |dfs.namenode.service.handler.count| *for the datanodes* it is default
> set to 10. Or you can configure this |dfs.namenode.handler.count| *for
> other clients*, like MapReduce jobs, |JobClients| that want to run a
> job. When a request comes in and it want to create a new handler, it
> go may out of memory (new Threads are also allocating a good chunk of
> stack space, maybe you need to increase this).
>
> I can’t find the dfs.namenode.service.handler.count in hdfs-site.xml
> of hadoop 2.3, is is deprecated? but I found it in the source.
>
> BTW, any suggestion for these parameters?
>
> thanks.
>
>
Re: “dfs.namenode.service.handler.count” and “dfs.namenode.handler.count”
Posted by ajisakaa <aj...@oss.nttdata.co.jp>.
Hi jason,
Looked around the source-code, both parameters are not deprecated.
dfs.namenode.service.handler.count : Specifies the number of
threads for NameNode to handle RPC requests from DataNodes,
standby NameNode, and all other non-client nodes
(BackupNode and SecondaryNameNode). Default to 10.
This parameter is valid only if dfs.namenode.servicerpc-address
is configured. If the parameter is configured, NameNode
initializes extra RPC server to handle requests from
non-client nodes.
dfs.namenode.handler.count : Specifies the number of threads
for NameNode to handle RPC requests. Default to 10.
I'll file a jira to document dfs.namenode.service.handler.count.
Regards,
Akira
On 05/20/2015 12:30 PM, jason lu wrote:
> Hi,
>
> what is the difference between “dfs.namenode.service.handler.count”
> and “dfs.namenode.handler.count” in hdfs-site.xml?
> I found this in stackoverflow:
> # The RPC Server needs threads to handle requests- Hadoop ships with his
> own RPC framework and you can configure this with the
> |dfs.namenode.service.handler.count| *for the datanodes* it is default
> set to 10. Or you can configure this |dfs.namenode.handler.count| *for
> other clients*, like MapReduce jobs, |JobClients| that want to run a
> job. When a request comes in and it want to create a new handler, it
> go may out of memory (new Threads are also allocating a good chunk of
> stack space, maybe you need to increase this).
>
> I can’t find the dfs.namenode.service.handler.count in hdfs-site.xml
> of hadoop 2.3, is is deprecated? but I found it in the source.
>
> BTW, any suggestion for these parameters?
>
> thanks.
>
>
Re: “dfs.namenode.service.handler.count” and “dfs.namenode.handler.count”
Posted by ajisakaa <aj...@oss.nttdata.co.jp>.
Hi jason,
Looked around the source-code, both parameters are not deprecated.
dfs.namenode.service.handler.count : Specifies the number of
threads for NameNode to handle RPC requests from DataNodes,
standby NameNode, and all other non-client nodes
(BackupNode and SecondaryNameNode). Default to 10.
This parameter is valid only if dfs.namenode.servicerpc-address
is configured. If the parameter is configured, NameNode
initializes extra RPC server to handle requests from
non-client nodes.
dfs.namenode.handler.count : Specifies the number of threads
for NameNode to handle RPC requests. Default to 10.
I'll file a jira to document dfs.namenode.service.handler.count.
Regards,
Akira
On 05/20/2015 12:30 PM, jason lu wrote:
> Hi,
>
> what is the difference between “dfs.namenode.service.handler.count”
> and “dfs.namenode.handler.count” in hdfs-site.xml?
> I found this in stackoverflow:
> # The RPC Server needs threads to handle requests- Hadoop ships with his
> own RPC framework and you can configure this with the
> |dfs.namenode.service.handler.count| *for the datanodes* it is default
> set to 10. Or you can configure this |dfs.namenode.handler.count| *for
> other clients*, like MapReduce jobs, |JobClients| that want to run a
> job. When a request comes in and it want to create a new handler, it
> go may out of memory (new Threads are also allocating a good chunk of
> stack space, maybe you need to increase this).
>
> I can’t find the dfs.namenode.service.handler.count in hdfs-site.xml
> of hadoop 2.3, is is deprecated? but I found it in the source.
>
> BTW, any suggestion for these parameters?
>
> thanks.
>
>