You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Yiqun Lin (JIRA)" <ji...@apache.org> on 2018/03/02 09:01:00 UTC
[jira] [Comment Edited] (HDFS-13214) RBF: Configuration on Router
conflicts with client side configuration
[ https://issues.apache.org/jira/browse/HDFS-13214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383371#comment-16383371 ]
Yiqun Lin edited comment on HDFS-13214 at 3/2/18 9:00 AM:
----------------------------------------------------------
Hi [~Tao Jie],
{quote}
So I think we should set dfs.nameservice.id to ns1 or ns2 rather than ns-fed.
{quote}
Yes, this should be right.
{quote}
Today property dfs.nameservice.id is not a necessary one in a federation cluster (HA or non-HA), right?
{quote}
From the function {{DFSUtil.getNameServiceId(Configuration conf, String addressKey)}} which used in the Router, {{dfs.nameservice.id}} doesn't must be specified. If this key isn't set, then it find the nameservice Id by matching the addressKey with the the address of the local node. If multiple addresses that matched, then the error will be thrown.
To avoid some potential problems happening in setting up Router, we can recommend users to configure the {{dfs.nameservice.id}} which help directly find the local node. If the local node is in a HA mode, {{dfs.ha.namenode.id}} is also recommended to configure. Otherwise, it will also found this by matching way. This can be documented in the doc of RBF. Any other thoughts?
was (Author: linyiqun):
Hi [~Tao Jie],
{quote}
So I think we should set dfs.nameservice.id to ns1 or ns2 rather than ns-fed.
{quote}
Yes, this should be right.
{quote}
Today property dfs.nameservice.id is not a necessary one in a federation cluster (HA or non-HA), right?
{quote}
From the function {{DFSUtil.getNameServiceId(Configuration conf, String addressKey)}} which used in the Router, {{dfs.nameservice.id}} doesn't must be specified. If this key isn't set, then it find the nameservice Id by matching the addressKey with the the address of the local node. If multiple addresses that matched, then the error will be thrown.
To avoid some potential problems happening in setting up Router, we can recommend users to configure the {{dfs.nameservice.id}}. If the local node is in a HA mode, {{dfs.ha.namenode.id}} is also recommended to configure. Otherwise, it will also found this by matching way. This can be documented in the doc of RBF. Any other thoughts?
> RBF: Configuration on Router conflicts with client side configuration
> ---------------------------------------------------------------------
>
> Key: HDFS-13214
> URL: https://issues.apache.org/jira/browse/HDFS-13214
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: 2.9.0
> Reporter: Tao Jie
> Priority: Major
>
> In a typical router-based federation cluster, hdfs-site.xml is supposed to be:
> {code}
> <property>
> <name>dfs.nameservices</name>
> <value>ns1,ns2,ns-fed</value>
> </property>
> <property>
> <name>dfs.ha.namenodes.ns-fed</name>
> <value>r1,r2</value>
> </property>
> <property>
> <name>dfs.namenode.rpc-address.ns1</name>
> <value>host1:8020</value>
> </property>
> <property>
> <name>dfs.namenode.rpc-address.ns2</name>
> <value>host2:8020</value>
> </property>
> <property>
> <name>dfs.namenode.rpc-address.ns-fed.r1</name>
> <value>host1:8888</value>
> </property>
> <property>
> <name>dfs.namenode.rpc-address.ns-fed.r2</name>
> <value>host2:8888</value>
> </property>
> {code}
> {{dfs.ha.namenodes.ns-fed}} here is used for client to access the Router. However with this configuration on server node, Router fails to start with error:
> {code}
> org.apache.hadoop.HadoopIllegalArgumentException: Configuration has multiple addresses that match local node's address. Please configure the system with dfs.nameservice.id and dfs.ha.namenode.id
> at org.apache.hadoop.hdfs.DFSUtil.getSuffixIDs(DFSUtil.java:1198)
> at org.apache.hadoop.hdfs.DFSUtil.getNameServiceId(DFSUtil.java:1131)
> at org.apache.hadoop.hdfs.DFSUtil.getNamenodeNameServiceId(DFSUtil.java:1086)
> at org.apache.hadoop.hdfs.server.federation.router.Router.createLocalNamenodeHearbeatService(Router.java:466)
> at org.apache.hadoop.hdfs.server.federation.router.Router.createNamenodeHearbeatServices(Router.java:423)
> at org.apache.hadoop.hdfs.server.federation.router.Router.serviceInit(Router.java:199)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at org.apache.hadoop.hdfs.server.federation.router.DFSRouter.main(DFSRouter.java:69)
> 2018-03-01 18:05:56,208 ERROR org.apache.hadoop.hdfs.server.federation.router.DFSRouter: Failed to start router
> {code}
> Then the router tries to find the local namenode, multiple properties: {{dfs.namenode.rpc-address.ns1}}, {{dfs.namenode.rpc-address.ns-fed.r1}} match the local address.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org