You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Bing Jiang <ji...@gmail.com> on 2013/07/04 09:12:38 UTC
Datanode support different Namespace
Hi, all
We try to use hadoop-2.0.5-alpha, using two namespaces, one is for hbase
cluster, and the other one is for common use.At the same time, we use
Quorum Journal policy as HA.
GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace
GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace.
GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode
After launching the hdfs cluster all, there is something which makes me
confused, that each namespace has half of the datanodes.
NameNode 'GS-CIX-SEV0004:9100'
Started: Thu Jul 04 10:28:00 CST 2013
Version: 2.0.5-alpha, 1488459
Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
Cluster ID: CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7
Block Pool ID: BP-1792015895-10.100.2.3-1372904504940
Browse the filesystem
NameNode Logs
Go back to DFS home
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Live Datanodes : 4
Last Admin Configured Used Non DFS Remaining
Used Used Remaining Block Block Pool Failed
Node Contact State Capacity (GB) Used (GB)
(%) (%) (%) Blocks Pool Used (%)> Volumes
(GB)
(GB) Used (GB) Blocks
GS-CIX-SEV0001 1 In Service 888.07 0.00 116.04 772.03 0.00
┌────────────┐ 86.93 0 0.00 0.00 0
└────────────┘
GS-CIX-SEV0002 1 In Service 888.07 0.00 135.50 752.57 0.00
┌────────────┐ 84.74 0 0.00 0.00 0
└────────────┘
GS-CIX-SEV0005 1 In Service 888.07 0.00 97.61 790.46 0.00
┌────────────┐ 89.01 0 0.00 0.00 0
└────────────┘
GS-CIX-SEV0006 1 In Service 888.07 0.00 122.30 765.77 0.00
┌────────────┐ 86.23 0 0.00 0.00 0
└────────────┘
Another Namespace's NameNode:
NameNode 'GS-CIX-SEV0001:9100'
Started: Thu Jul 04 10:19:03 CST 2013
Version: 2.0.5-alpha, 1488459
Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
Cluster ID: CID-1a53483d-000e-4726-aef1-f500bedb1df6
Block Pool ID: BP-1142418822-10.100.2.1-1372904314309
Browse the filesystem
NameNode Logs
Go back to DFS home
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Live Datanodes : 4
Last Admin Configured Used Non DFS Remaining
Used Used Remaining Block Block Pool Failed
Node Contact State Capacity (GB) Used (GB)
(%) (%) (%) Blocks Pool Used (%)> Volumes
(GB)
(GB) Used (GB) Blocks
GS-CIX-SEV0003 0 In Service 888.07 0.00 150.54 737.53 0.00
┌────────────┐ 83.05 0 0.00 0.00 0
└────────────┘
GS-CIX-SEV0004 0 In Service 888.07 0.00 177.22 710.85 0.00
┌────────────┐ 80.04 0 0.00 0.00 0
└────────────┘
GS-CIX-SEV0007 0 In Service 888.07 0.00 62.91 825.16 0.00
┌────────────┐ 92.92 0 0.00 0.00 0
└────────────┘
GS-CIX-SEV0008 0 In Service 888.07 0.00 125.25 762.82 0.00
┌────────────┐ 85.90 0 0.00 0.00 0
And check the DN(GS-CIX-SEV0001)'s log, it prints like this:
2013-07-04 10:34:51,699 FATAL
org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id
DS-1677272131-10.100.2.1-50010-1372905291690) service to GS-CIX-SEV0001/
10.100.2.1:9100
java.io.IOException: Inconsistent storage IDs. Name-node returned
DS811369792. Expecting DS-1677272131-10.100.2.1-50010-1372905291690
at
org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:731)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:308)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:632)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
at java.lang.Thread.run(Thread.java:662)
It is proved that one datanode has been required to attached to only one
namespace?
Any views about it will be thankful.
Regards~
--
Bing Jiang
Tel：(86)134-2619-1361
weibo: http://weibo.com/jiangbinglover
BLOG: http://blog.sina.com.cn/jiangbinglover
National Research Center for Intelligent Computing Systems
Institute of Computing technology
Graduate University of Chinese Academy of Science
Re: Datanode support different Namespace
Posted by Azuryy Yu <az...@gmail.com>.
It's random.
On Jul 4, 2013 3:33 PM, "Bing Jiang" <ji...@gmail.com> wrote:
> If not set cluster id in formatting the Namenode, is there a policy in
> hdfs to guarantee the even of distributing DataNodes into different
> Namespace, or just randomly?
>
>
>
> 2013/7/4 Azuryy Yu <az...@gmail.com>
>
>> Additional,
>>
>> If these are two new clusters, then on each namenode, using "hdfs
>> namenode -format -clusterID yourID"
>>
>> But if you want to upgrade these two clusters from NonHA to HA, then
>> using "bin/start-dfs.sh -upgrade -clusterID yourID"
>>
>>
>>
>> On Thu, Jul 4, 2013 at 3:14 PM, Azuryy Yu <az...@gmail.com> wrote:
>>
>>>
>>> This is because you don't use the same clusterID. all data nodes and
>>> namenodes should use the same clusterID.
>>>
>>>
>>> On Thu, Jul 4, 2013 at 3:12 PM, Bing Jiang <ji...@gmail.com>wrote:
>>>
>>>> Hi, all
>>>>
>>>> We try to use hadoop-2.0.5-alpha, using two namespaces, one is for
>>>> hbase cluster, and the other one is for common use.At the same time, we use
>>>> Quorum Journal policy as HA.
>>>>
>>>> GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace
>>>>
>>>> GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace.
>>>>
>>>> GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode
>>>>
>>>> After launching the hdfs cluster all, there is something which makes
>>>> me confused, that each namespace has half of the datanodes.
>>>>
>>>> NameNode 'GS-CIX-SEV0004:9100'
>>>>
>>>> Started: Thu Jul 04 10:28:00 CST 2013
>>>> Version: 2.0.5-alpha, 1488459
>>>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>>>> Cluster ID: CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7
>>>> Block Pool ID: BP-1792015895-10.100.2.3-1372904504940
>>>>
>>>>
>>>> Browse the filesystem
>>>> NameNode Logs
>>>> Go back to DFS home
>>>>
>>>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>>>> Live Datanodes : 4
>>>>
>>>>
>>>> Last Admin Configured Used Non DFS Remaining
>>>> Used Used Remaining Block Block Pool Failed
>>>> Node Contact State Capacity (GB) Used (GB)
>>>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>>>> (GB)
>>>> (GB) Used (GB) Blocks
>>>> GS-CIX-SEV0001 1 In Service 888.07 0.00 116.04 772.03
>>>> 0.00 ┌────────────┐ 86.93 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0002 1 In Service 888.07 0.00 135.50 752.57
>>>> 0.00 ┌────────────┐ 84.74 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0005 1 In Service 888.07 0.00 97.61 790.46
>>>> 0.00 ┌────────────┐ 89.01 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0006 1 In Service 888.07 0.00 122.30 765.77
>>>> 0.00 ┌────────────┐ 86.23 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>>
>>>>
>>>> Another Namespace's NameNode:
>>>>
>>>> NameNode 'GS-CIX-SEV0001:9100'
>>>>
>>>> Started: Thu Jul 04 10:19:03 CST 2013
>>>> Version: 2.0.5-alpha, 1488459
>>>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>>>> Cluster ID: CID-1a53483d-000e-4726-aef1-f500bedb1df6
>>>> Block Pool ID: BP-1142418822-10.100.2.1-1372904314309
>>>>
>>>>
>>>> Browse the filesystem
>>>> NameNode Logs
>>>> Go back to DFS home
>>>>
>>>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>>>> Live Datanodes : 4
>>>>
>>>>
>>>> Last Admin Configured Used Non DFS Remaining
>>>> Used Used Remaining Block Block Pool Failed
>>>> Node Contact State Capacity (GB) Used (GB)
>>>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>>>> (GB)
>>>> (GB) Used (GB) Blocks
>>>> GS-CIX-SEV0003 0 In Service 888.07 0.00 150.54 737.53
>>>> 0.00 ┌────────────┐ 83.05 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0004 0 In Service 888.07 0.00 177.22 710.85
>>>> 0.00 ┌────────────┐ 80.04 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0007 0 In Service 888.07 0.00 62.91 825.16
>>>> 0.00 ┌────────────┐ 92.92 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0008 0 In Service 888.07 0.00 125.25 762.82
>>>> 0.00 ┌────────────┐ 85.90 0 0.00 0.00 0
>>>>
>>>>
>>>>
>>>> And check the DN(GS-CIX-SEV0001)'s log, it prints like this:
>>>> 2013-07-04 10:34:51,699 FATAL
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
>>>> block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id
>>>> DS-1677272131-10.100.2.1-50010-1372905291690) service to GS-CIX-SEV0001/
>>>> 10.100.2.1:9100
>>>> java.io.IOException: Inconsistent storage IDs. Name-node returned
>>>> DS811369792. Expecting DS-1677272131-10.100.2.1-50010-1372905291690
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:731)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:308)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:632)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> It is proved that one datanode has been required to attached to only
>>>> one namespace?
>>>>
>>>> Any views about it will be thankful.
>>>>
>>>> Regards~
>>>>
>>>>
>>>> --
>>>> Bing Jiang
>>>> Tel：(86)134-2619-1361
>>>> weibo: http://weibo.com/jiangbinglover
>>>> BLOG: http://blog.sina.com.cn/jiangbinglover
>>>> National Research Center for Intelligent Computing Systems
>>>> Institute of Computing technology
>>>> Graduate University of Chinese Academy of Science
>>>>
>>>
>>>
>>
>
>
> --
> Bing Jiang
> Tel：(86)134-2619-1361
> weibo: http://weibo.com/jiangbinglover
> BLOG: http://blog.sina.com.cn/jiangbinglover
> National Research Center for Intelligent Computing Systems
> Institute of Computing technology
> Graduate University of Chinese Academy of Science
>
Re: Datanode support different Namespace
Posted by Azuryy Yu <az...@gmail.com>.
It's random.
On Jul 4, 2013 3:33 PM, "Bing Jiang" <ji...@gmail.com> wrote:
> If not set cluster id in formatting the Namenode, is there a policy in
> hdfs to guarantee the even of distributing DataNodes into different
> Namespace, or just randomly?
>
>
>
> 2013/7/4 Azuryy Yu <az...@gmail.com>
>
>> Additional,
>>
>> If these are two new clusters, then on each namenode, using "hdfs
>> namenode -format -clusterID yourID"
>>
>> But if you want to upgrade these two clusters from NonHA to HA, then
>> using "bin/start-dfs.sh -upgrade -clusterID yourID"
>>
>>
>>
>> On Thu, Jul 4, 2013 at 3:14 PM, Azuryy Yu <az...@gmail.com> wrote:
>>
>>>
>>> This is because you don't use the same clusterID. all data nodes and
>>> namenodes should use the same clusterID.
>>>
>>>
>>> On Thu, Jul 4, 2013 at 3:12 PM, Bing Jiang <ji...@gmail.com>wrote:
>>>
>>>> Hi, all
>>>>
>>>> We try to use hadoop-2.0.5-alpha, using two namespaces, one is for
>>>> hbase cluster, and the other one is for common use.At the same time, we use
>>>> Quorum Journal policy as HA.
>>>>
>>>> GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace
>>>>
>>>> GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace.
>>>>
>>>> GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode
>>>>
>>>> After launching the hdfs cluster all, there is something which makes
>>>> me confused, that each namespace has half of the datanodes.
>>>>
>>>> NameNode 'GS-CIX-SEV0004:9100'
>>>>
>>>> Started: Thu Jul 04 10:28:00 CST 2013
>>>> Version: 2.0.5-alpha, 1488459
>>>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>>>> Cluster ID: CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7
>>>> Block Pool ID: BP-1792015895-10.100.2.3-1372904504940
>>>>
>>>>
>>>> Browse the filesystem
>>>> NameNode Logs
>>>> Go back to DFS home
>>>>
>>>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>>>> Live Datanodes : 4
>>>>
>>>>
>>>> Last Admin Configured Used Non DFS Remaining
>>>> Used Used Remaining Block Block Pool Failed
>>>> Node Contact State Capacity (GB) Used (GB)
>>>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>>>> (GB)
>>>> (GB) Used (GB) Blocks
>>>> GS-CIX-SEV0001 1 In Service 888.07 0.00 116.04 772.03
>>>> 0.00 ┌────────────┐ 86.93 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0002 1 In Service 888.07 0.00 135.50 752.57
>>>> 0.00 ┌────────────┐ 84.74 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0005 1 In Service 888.07 0.00 97.61 790.46
>>>> 0.00 ┌────────────┐ 89.01 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0006 1 In Service 888.07 0.00 122.30 765.77
>>>> 0.00 ┌────────────┐ 86.23 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>>
>>>>
>>>> Another Namespace's NameNode:
>>>>
>>>> NameNode 'GS-CIX-SEV0001:9100'
>>>>
>>>> Started: Thu Jul 04 10:19:03 CST 2013
>>>> Version: 2.0.5-alpha, 1488459
>>>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>>>> Cluster ID: CID-1a53483d-000e-4726-aef1-f500bedb1df6
>>>> Block Pool ID: BP-1142418822-10.100.2.1-1372904314309
>>>>
>>>>
>>>> Browse the filesystem
>>>> NameNode Logs
>>>> Go back to DFS home
>>>>
>>>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>>>> Live Datanodes : 4
>>>>
>>>>
>>>> Last Admin Configured Used Non DFS Remaining
>>>> Used Used Remaining Block Block Pool Failed
>>>> Node Contact State Capacity (GB) Used (GB)
>>>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>>>> (GB)
>>>> (GB) Used (GB) Blocks
>>>> GS-CIX-SEV0003 0 In Service 888.07 0.00 150.54 737.53
>>>> 0.00 ┌────────────┐ 83.05 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0004 0 In Service 888.07 0.00 177.22 710.85
>>>> 0.00 ┌────────────┐ 80.04 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0007 0 In Service 888.07 0.00 62.91 825.16
>>>> 0.00 ┌────────────┐ 92.92 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0008 0 In Service 888.07 0.00 125.25 762.82
>>>> 0.00 ┌────────────┐ 85.90 0 0.00 0.00 0
>>>>
>>>>
>>>>
>>>> And check the DN(GS-CIX-SEV0001)'s log, it prints like this:
>>>> 2013-07-04 10:34:51,699 FATAL
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
>>>> block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id
>>>> DS-1677272131-10.100.2.1-50010-1372905291690) service to GS-CIX-SEV0001/
>>>> 10.100.2.1:9100
>>>> java.io.IOException: Inconsistent storage IDs. Name-node returned
>>>> DS811369792. Expecting DS-1677272131-10.100.2.1-50010-1372905291690
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:731)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:308)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:632)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> It is proved that one datanode has been required to attached to only
>>>> one namespace?
>>>>
>>>> Any views about it will be thankful.
>>>>
>>>> Regards~
>>>>
>>>>
>>>> --
>>>> Bing Jiang
>>>> Tel：(86)134-2619-1361
>>>> weibo: http://weibo.com/jiangbinglover
>>>> BLOG: http://blog.sina.com.cn/jiangbinglover
>>>> National Research Center for Intelligent Computing Systems
>>>> Institute of Computing technology
>>>> Graduate University of Chinese Academy of Science
>>>>
>>>
>>>
>>
>
>
> --
> Bing Jiang
> Tel：(86)134-2619-1361
> weibo: http://weibo.com/jiangbinglover
> BLOG: http://blog.sina.com.cn/jiangbinglover
> National Research Center for Intelligent Computing Systems
> Institute of Computing technology
> Graduate University of Chinese Academy of Science
>
Re: Datanode support different Namespace
Posted by Azuryy Yu <az...@gmail.com>.
It's random.
On Jul 4, 2013 3:33 PM, "Bing Jiang" <ji...@gmail.com> wrote:
> If not set cluster id in formatting the Namenode, is there a policy in
> hdfs to guarantee the even of distributing DataNodes into different
> Namespace, or just randomly?
>
>
>
> 2013/7/4 Azuryy Yu <az...@gmail.com>
>
>> Additional,
>>
>> If these are two new clusters, then on each namenode, using "hdfs
>> namenode -format -clusterID yourID"
>>
>> But if you want to upgrade these two clusters from NonHA to HA, then
>> using "bin/start-dfs.sh -upgrade -clusterID yourID"
>>
>>
>>
>> On Thu, Jul 4, 2013 at 3:14 PM, Azuryy Yu <az...@gmail.com> wrote:
>>
>>>
>>> This is because you don't use the same clusterID. all data nodes and
>>> namenodes should use the same clusterID.
>>>
>>>
>>> On Thu, Jul 4, 2013 at 3:12 PM, Bing Jiang <ji...@gmail.com>wrote:
>>>
>>>> Hi, all
>>>>
>>>> We try to use hadoop-2.0.5-alpha, using two namespaces, one is for
>>>> hbase cluster, and the other one is for common use.At the same time, we use
>>>> Quorum Journal policy as HA.
>>>>
>>>> GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace
>>>>
>>>> GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace.
>>>>
>>>> GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode
>>>>
>>>> After launching the hdfs cluster all, there is something which makes
>>>> me confused, that each namespace has half of the datanodes.
>>>>
>>>> NameNode 'GS-CIX-SEV0004:9100'
>>>>
>>>> Started: Thu Jul 04 10:28:00 CST 2013
>>>> Version: 2.0.5-alpha, 1488459
>>>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>>>> Cluster ID: CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7
>>>> Block Pool ID: BP-1792015895-10.100.2.3-1372904504940
>>>>
>>>>
>>>> Browse the filesystem
>>>> NameNode Logs
>>>> Go back to DFS home
>>>>
>>>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>>>> Live Datanodes : 4
>>>>
>>>>
>>>> Last Admin Configured Used Non DFS Remaining
>>>> Used Used Remaining Block Block Pool Failed
>>>> Node Contact State Capacity (GB) Used (GB)
>>>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>>>> (GB)
>>>> (GB) Used (GB) Blocks
>>>> GS-CIX-SEV0001 1 In Service 888.07 0.00 116.04 772.03
>>>> 0.00 ┌────────────┐ 86.93 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0002 1 In Service 888.07 0.00 135.50 752.57
>>>> 0.00 ┌────────────┐ 84.74 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0005 1 In Service 888.07 0.00 97.61 790.46
>>>> 0.00 ┌────────────┐ 89.01 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0006 1 In Service 888.07 0.00 122.30 765.77
>>>> 0.00 ┌────────────┐ 86.23 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>>
>>>>
>>>> Another Namespace's NameNode:
>>>>
>>>> NameNode 'GS-CIX-SEV0001:9100'
>>>>
>>>> Started: Thu Jul 04 10:19:03 CST 2013
>>>> Version: 2.0.5-alpha, 1488459
>>>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>>>> Cluster ID: CID-1a53483d-000e-4726-aef1-f500bedb1df6
>>>> Block Pool ID: BP-1142418822-10.100.2.1-1372904314309
>>>>
>>>>
>>>> Browse the filesystem
>>>> NameNode Logs
>>>> Go back to DFS home
>>>>
>>>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>>>> Live Datanodes : 4
>>>>
>>>>
>>>> Last Admin Configured Used Non DFS Remaining
>>>> Used Used Remaining Block Block Pool Failed
>>>> Node Contact State Capacity (GB) Used (GB)
>>>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>>>> (GB)
>>>> (GB) Used (GB) Blocks
>>>> GS-CIX-SEV0003 0 In Service 888.07 0.00 150.54 737.53
>>>> 0.00 ┌────────────┐ 83.05 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0004 0 In Service 888.07 0.00 177.22 710.85
>>>> 0.00 ┌────────────┐ 80.04 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0007 0 In Service 888.07 0.00 62.91 825.16
>>>> 0.00 ┌────────────┐ 92.92 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0008 0 In Service 888.07 0.00 125.25 762.82
>>>> 0.00 ┌────────────┐ 85.90 0 0.00 0.00 0
>>>>
>>>>
>>>>
>>>> And check the DN(GS-CIX-SEV0001)'s log, it prints like this:
>>>> 2013-07-04 10:34:51,699 FATAL
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
>>>> block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id
>>>> DS-1677272131-10.100.2.1-50010-1372905291690) service to GS-CIX-SEV0001/
>>>> 10.100.2.1:9100
>>>> java.io.IOException: Inconsistent storage IDs. Name-node returned
>>>> DS811369792. Expecting DS-1677272131-10.100.2.1-50010-1372905291690
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:731)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:308)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:632)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> It is proved that one datanode has been required to attached to only
>>>> one namespace?
>>>>
>>>> Any views about it will be thankful.
>>>>
>>>> Regards~
>>>>
>>>>
>>>> --
>>>> Bing Jiang
>>>> Tel：(86)134-2619-1361
>>>> weibo: http://weibo.com/jiangbinglover
>>>> BLOG: http://blog.sina.com.cn/jiangbinglover
>>>> National Research Center for Intelligent Computing Systems
>>>> Institute of Computing technology
>>>> Graduate University of Chinese Academy of Science
>>>>
>>>
>>>
>>
>
>
> --
> Bing Jiang
> Tel：(86)134-2619-1361
> weibo: http://weibo.com/jiangbinglover
> BLOG: http://blog.sina.com.cn/jiangbinglover
> National Research Center for Intelligent Computing Systems
> Institute of Computing technology
> Graduate University of Chinese Academy of Science
>
Re: Datanode support different Namespace
Posted by Azuryy Yu <az...@gmail.com>.
It's random.
On Jul 4, 2013 3:33 PM, "Bing Jiang" <ji...@gmail.com> wrote:
> If not set cluster id in formatting the Namenode, is there a policy in
> hdfs to guarantee the even of distributing DataNodes into different
> Namespace, or just randomly?
>
>
>
> 2013/7/4 Azuryy Yu <az...@gmail.com>
>
>> Additional,
>>
>> If these are two new clusters, then on each namenode, using "hdfs
>> namenode -format -clusterID yourID"
>>
>> But if you want to upgrade these two clusters from NonHA to HA, then
>> using "bin/start-dfs.sh -upgrade -clusterID yourID"
>>
>>
>>
>> On Thu, Jul 4, 2013 at 3:14 PM, Azuryy Yu <az...@gmail.com> wrote:
>>
>>>
>>> This is because you don't use the same clusterID. all data nodes and
>>> namenodes should use the same clusterID.
>>>
>>>
>>> On Thu, Jul 4, 2013 at 3:12 PM, Bing Jiang <ji...@gmail.com>wrote:
>>>
>>>> Hi, all
>>>>
>>>> We try to use hadoop-2.0.5-alpha, using two namespaces, one is for
>>>> hbase cluster, and the other one is for common use.At the same time, we use
>>>> Quorum Journal policy as HA.
>>>>
>>>> GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace
>>>>
>>>> GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace.
>>>>
>>>> GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode
>>>>
>>>> After launching the hdfs cluster all, there is something which makes
>>>> me confused, that each namespace has half of the datanodes.
>>>>
>>>> NameNode 'GS-CIX-SEV0004:9100'
>>>>
>>>> Started: Thu Jul 04 10:28:00 CST 2013
>>>> Version: 2.0.5-alpha, 1488459
>>>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>>>> Cluster ID: CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7
>>>> Block Pool ID: BP-1792015895-10.100.2.3-1372904504940
>>>>
>>>>
>>>> Browse the filesystem
>>>> NameNode Logs
>>>> Go back to DFS home
>>>>
>>>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>>>> Live Datanodes : 4
>>>>
>>>>
>>>> Last Admin Configured Used Non DFS Remaining
>>>> Used Used Remaining Block Block Pool Failed
>>>> Node Contact State Capacity (GB) Used (GB)
>>>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>>>> (GB)
>>>> (GB) Used (GB) Blocks
>>>> GS-CIX-SEV0001 1 In Service 888.07 0.00 116.04 772.03
>>>> 0.00 ┌────────────┐ 86.93 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0002 1 In Service 888.07 0.00 135.50 752.57
>>>> 0.00 ┌────────────┐ 84.74 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0005 1 In Service 888.07 0.00 97.61 790.46
>>>> 0.00 ┌────────────┐ 89.01 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0006 1 In Service 888.07 0.00 122.30 765.77
>>>> 0.00 ┌────────────┐ 86.23 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>>
>>>>
>>>> Another Namespace's NameNode:
>>>>
>>>> NameNode 'GS-CIX-SEV0001:9100'
>>>>
>>>> Started: Thu Jul 04 10:19:03 CST 2013
>>>> Version: 2.0.5-alpha, 1488459
>>>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>>>> Cluster ID: CID-1a53483d-000e-4726-aef1-f500bedb1df6
>>>> Block Pool ID: BP-1142418822-10.100.2.1-1372904314309
>>>>
>>>>
>>>> Browse the filesystem
>>>> NameNode Logs
>>>> Go back to DFS home
>>>>
>>>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>>>> Live Datanodes : 4
>>>>
>>>>
>>>> Last Admin Configured Used Non DFS Remaining
>>>> Used Used Remaining Block Block Pool Failed
>>>> Node Contact State Capacity (GB) Used (GB)
>>>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>>>> (GB)
>>>> (GB) Used (GB) Blocks
>>>> GS-CIX-SEV0003 0 In Service 888.07 0.00 150.54 737.53
>>>> 0.00 ┌────────────┐ 83.05 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0004 0 In Service 888.07 0.00 177.22 710.85
>>>> 0.00 ┌────────────┐ 80.04 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0007 0 In Service 888.07 0.00 62.91 825.16
>>>> 0.00 ┌────────────┐ 92.92 0 0.00 0.00 0
>>>>
>>>> └────────────┘
>>>> GS-CIX-SEV0008 0 In Service 888.07 0.00 125.25 762.82
>>>> 0.00 ┌────────────┐ 85.90 0 0.00 0.00 0
>>>>
>>>>
>>>>
>>>> And check the DN(GS-CIX-SEV0001)'s log, it prints like this:
>>>> 2013-07-04 10:34:51,699 FATAL
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
>>>> block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id
>>>> DS-1677272131-10.100.2.1-50010-1372905291690) service to GS-CIX-SEV0001/
>>>> 10.100.2.1:9100
>>>> java.io.IOException: Inconsistent storage IDs. Name-node returned
>>>> DS811369792. Expecting DS-1677272131-10.100.2.1-50010-1372905291690
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:731)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:308)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:632)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> It is proved that one datanode has been required to attached to only
>>>> one namespace?
>>>>
>>>> Any views about it will be thankful.
>>>>
>>>> Regards~
>>>>
>>>>
>>>> --
>>>> Bing Jiang
>>>> Tel：(86)134-2619-1361
>>>> weibo: http://weibo.com/jiangbinglover
>>>> BLOG: http://blog.sina.com.cn/jiangbinglover
>>>> National Research Center for Intelligent Computing Systems
>>>> Institute of Computing technology
>>>> Graduate University of Chinese Academy of Science
>>>>
>>>
>>>
>>
>
>
> --
> Bing Jiang
> Tel：(86)134-2619-1361
> weibo: http://weibo.com/jiangbinglover
> BLOG: http://blog.sina.com.cn/jiangbinglover
> National Research Center for Intelligent Computing Systems
> Institute of Computing technology
> Graduate University of Chinese Academy of Science
>
Re: Datanode support different Namespace
Posted by Bing Jiang <ji...@gmail.com>.
If not set cluster id in formatting the Namenode, is there a policy in hdfs
to guarantee the even of distributing DataNodes into different Namespace,
or just randomly?
2013/7/4 Azuryy Yu <az...@gmail.com>
> Additional,
>
> If these are two new clusters, then on each namenode, using "hdfs namenode
> -format -clusterID yourID"
>
> But if you want to upgrade these two clusters from NonHA to HA, then using
> "bin/start-dfs.sh -upgrade -clusterID yourID"
>
>
>
> On Thu, Jul 4, 2013 at 3:14 PM, Azuryy Yu <az...@gmail.com> wrote:
>
>>
>> This is because you don't use the same clusterID. all data nodes and
>> namenodes should use the same clusterID.
>>
>>
>> On Thu, Jul 4, 2013 at 3:12 PM, Bing Jiang <ji...@gmail.com>wrote:
>>
>>> Hi, all
>>>
>>> We try to use hadoop-2.0.5-alpha, using two namespaces, one is for hbase
>>> cluster, and the other one is for common use.At the same time, we use
>>> Quorum Journal policy as HA.
>>>
>>> GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace
>>>
>>> GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace.
>>>
>>> GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode
>>>
>>> After launching the hdfs cluster all, there is something which makes me
>>> confused, that each namespace has half of the datanodes.
>>>
>>> NameNode 'GS-CIX-SEV0004:9100'
>>>
>>> Started: Thu Jul 04 10:28:00 CST 2013
>>> Version: 2.0.5-alpha, 1488459
>>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>>> Cluster ID: CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7
>>> Block Pool ID: BP-1792015895-10.100.2.3-1372904504940
>>>
>>>
>>> Browse the filesystem
>>> NameNode Logs
>>> Go back to DFS home
>>>
>>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>>> Live Datanodes : 4
>>>
>>>
>>> Last Admin Configured Used Non DFS Remaining
>>> Used Used Remaining Block Block Pool Failed
>>> Node Contact State Capacity (GB) Used (GB)
>>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>>> (GB)
>>> (GB) Used (GB) Blocks
>>> GS-CIX-SEV0001 1 In Service 888.07 0.00 116.04 772.03
>>> 0.00 ┌────────────┐ 86.93 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0002 1 In Service 888.07 0.00 135.50 752.57
>>> 0.00 ┌────────────┐ 84.74 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0005 1 In Service 888.07 0.00 97.61 790.46
>>> 0.00 ┌────────────┐ 89.01 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0006 1 In Service 888.07 0.00 122.30 765.77
>>> 0.00 ┌────────────┐ 86.23 0 0.00 0.00 0
>>>
>>> └────────────┘
>>>
>>>
>>> Another Namespace's NameNode:
>>>
>>> NameNode 'GS-CIX-SEV0001:9100'
>>>
>>> Started: Thu Jul 04 10:19:03 CST 2013
>>> Version: 2.0.5-alpha, 1488459
>>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>>> Cluster ID: CID-1a53483d-000e-4726-aef1-f500bedb1df6
>>> Block Pool ID: BP-1142418822-10.100.2.1-1372904314309
>>>
>>>
>>> Browse the filesystem
>>> NameNode Logs
>>> Go back to DFS home
>>>
>>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>>> Live Datanodes : 4
>>>
>>>
>>> Last Admin Configured Used Non DFS Remaining
>>> Used Used Remaining Block Block Pool Failed
>>> Node Contact State Capacity (GB) Used (GB)
>>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>>> (GB)
>>> (GB) Used (GB) Blocks
>>> GS-CIX-SEV0003 0 In Service 888.07 0.00 150.54 737.53
>>> 0.00 ┌────────────┐ 83.05 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0004 0 In Service 888.07 0.00 177.22 710.85
>>> 0.00 ┌────────────┐ 80.04 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0007 0 In Service 888.07 0.00 62.91 825.16
>>> 0.00 ┌────────────┐ 92.92 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0008 0 In Service 888.07 0.00 125.25 762.82
>>> 0.00 ┌────────────┐ 85.90 0 0.00 0.00 0
>>>
>>>
>>>
>>> And check the DN(GS-CIX-SEV0001)'s log, it prints like this:
>>> 2013-07-04 10:34:51,699 FATAL
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
>>> block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id
>>> DS-1677272131-10.100.2.1-50010-1372905291690) service to GS-CIX-SEV0001/
>>> 10.100.2.1:9100
>>> java.io.IOException: Inconsistent storage IDs. Name-node returned
>>> DS811369792. Expecting DS-1677272131-10.100.2.1-50010-1372905291690
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:731)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:308)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:632)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> It is proved that one datanode has been required to attached to only one
>>> namespace?
>>>
>>> Any views about it will be thankful.
>>>
>>> Regards~
>>>
>>>
>>> --
>>> Bing Jiang
>>> Tel：(86)134-2619-1361
>>> weibo: http://weibo.com/jiangbinglover
>>> BLOG: http://blog.sina.com.cn/jiangbinglover
>>> National Research Center for Intelligent Computing Systems
>>> Institute of Computing technology
>>> Graduate University of Chinese Academy of Science
>>>
>>
>>
>
--
Bing Jiang
Tel：(86)134-2619-1361
weibo: http://weibo.com/jiangbinglover
BLOG: http://blog.sina.com.cn/jiangbinglover
National Research Center for Intelligent Computing Systems
Institute of Computing technology
Graduate University of Chinese Academy of Science
Re: Datanode support different Namespace
Posted by Bing Jiang <ji...@gmail.com>.
If not set cluster id in formatting the Namenode, is there a policy in hdfs
to guarantee the even of distributing DataNodes into different Namespace,
or just randomly?
2013/7/4 Azuryy Yu <az...@gmail.com>
> Additional,
>
> If these are two new clusters, then on each namenode, using "hdfs namenode
> -format -clusterID yourID"
>
> But if you want to upgrade these two clusters from NonHA to HA, then using
> "bin/start-dfs.sh -upgrade -clusterID yourID"
>
>
>
> On Thu, Jul 4, 2013 at 3:14 PM, Azuryy Yu <az...@gmail.com> wrote:
>
>>
>> This is because you don't use the same clusterID. all data nodes and
>> namenodes should use the same clusterID.
>>
>>
>> On Thu, Jul 4, 2013 at 3:12 PM, Bing Jiang <ji...@gmail.com>wrote:
>>
>>> Hi, all
>>>
>>> We try to use hadoop-2.0.5-alpha, using two namespaces, one is for hbase
>>> cluster, and the other one is for common use.At the same time, we use
>>> Quorum Journal policy as HA.
>>>
>>> GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace
>>>
>>> GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace.
>>>
>>> GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode
>>>
>>> After launching the hdfs cluster all, there is something which makes me
>>> confused, that each namespace has half of the datanodes.
>>>
>>> NameNode 'GS-CIX-SEV0004:9100'
>>>
>>> Started: Thu Jul 04 10:28:00 CST 2013
>>> Version: 2.0.5-alpha, 1488459
>>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>>> Cluster ID: CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7
>>> Block Pool ID: BP-1792015895-10.100.2.3-1372904504940
>>>
>>>
>>> Browse the filesystem
>>> NameNode Logs
>>> Go back to DFS home
>>>
>>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>>> Live Datanodes : 4
>>>
>>>
>>> Last Admin Configured Used Non DFS Remaining
>>> Used Used Remaining Block Block Pool Failed
>>> Node Contact State Capacity (GB) Used (GB)
>>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>>> (GB)
>>> (GB) Used (GB) Blocks
>>> GS-CIX-SEV0001 1 In Service 888.07 0.00 116.04 772.03
>>> 0.00 ┌────────────┐ 86.93 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0002 1 In Service 888.07 0.00 135.50 752.57
>>> 0.00 ┌────────────┐ 84.74 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0005 1 In Service 888.07 0.00 97.61 790.46
>>> 0.00 ┌────────────┐ 89.01 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0006 1 In Service 888.07 0.00 122.30 765.77
>>> 0.00 ┌────────────┐ 86.23 0 0.00 0.00 0
>>>
>>> └────────────┘
>>>
>>>
>>> Another Namespace's NameNode:
>>>
>>> NameNode 'GS-CIX-SEV0001:9100'
>>>
>>> Started: Thu Jul 04 10:19:03 CST 2013
>>> Version: 2.0.5-alpha, 1488459
>>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>>> Cluster ID: CID-1a53483d-000e-4726-aef1-f500bedb1df6
>>> Block Pool ID: BP-1142418822-10.100.2.1-1372904314309
>>>
>>>
>>> Browse the filesystem
>>> NameNode Logs
>>> Go back to DFS home
>>>
>>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>>> Live Datanodes : 4
>>>
>>>
>>> Last Admin Configured Used Non DFS Remaining
>>> Used Used Remaining Block Block Pool Failed
>>> Node Contact State Capacity (GB) Used (GB)
>>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>>> (GB)
>>> (GB) Used (GB) Blocks
>>> GS-CIX-SEV0003 0 In Service 888.07 0.00 150.54 737.53
>>> 0.00 ┌────────────┐ 83.05 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0004 0 In Service 888.07 0.00 177.22 710.85
>>> 0.00 ┌────────────┐ 80.04 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0007 0 In Service 888.07 0.00 62.91 825.16
>>> 0.00 ┌────────────┐ 92.92 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0008 0 In Service 888.07 0.00 125.25 762.82
>>> 0.00 ┌────────────┐ 85.90 0 0.00 0.00 0
>>>
>>>
>>>
>>> And check the DN(GS-CIX-SEV0001)'s log, it prints like this:
>>> 2013-07-04 10:34:51,699 FATAL
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
>>> block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id
>>> DS-1677272131-10.100.2.1-50010-1372905291690) service to GS-CIX-SEV0001/
>>> 10.100.2.1:9100
>>> java.io.IOException: Inconsistent storage IDs. Name-node returned
>>> DS811369792. Expecting DS-1677272131-10.100.2.1-50010-1372905291690
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:731)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:308)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:632)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> It is proved that one datanode has been required to attached to only one
>>> namespace?
>>>
>>> Any views about it will be thankful.
>>>
>>> Regards~
>>>
>>>
>>> --
>>> Bing Jiang
>>> Tel：(86)134-2619-1361
>>> weibo: http://weibo.com/jiangbinglover
>>> BLOG: http://blog.sina.com.cn/jiangbinglover
>>> National Research Center for Intelligent Computing Systems
>>> Institute of Computing technology
>>> Graduate University of Chinese Academy of Science
>>>
>>
>>
>
--
Bing Jiang
Tel：(86)134-2619-1361
weibo: http://weibo.com/jiangbinglover
BLOG: http://blog.sina.com.cn/jiangbinglover
National Research Center for Intelligent Computing Systems
Institute of Computing technology
Graduate University of Chinese Academy of Science
Re: Datanode support different Namespace
Posted by Bing Jiang <ji...@gmail.com>.
If not set cluster id in formatting the Namenode, is there a policy in hdfs
to guarantee the even of distributing DataNodes into different Namespace,
or just randomly?
2013/7/4 Azuryy Yu <az...@gmail.com>
> Additional,
>
> If these are two new clusters, then on each namenode, using "hdfs namenode
> -format -clusterID yourID"
>
> But if you want to upgrade these two clusters from NonHA to HA, then using
> "bin/start-dfs.sh -upgrade -clusterID yourID"
>
>
>
> On Thu, Jul 4, 2013 at 3:14 PM, Azuryy Yu <az...@gmail.com> wrote:
>
>>
>> This is because you don't use the same clusterID. all data nodes and
>> namenodes should use the same clusterID.
>>
>>
>> On Thu, Jul 4, 2013 at 3:12 PM, Bing Jiang <ji...@gmail.com>wrote:
>>
>>> Hi, all
>>>
>>> We try to use hadoop-2.0.5-alpha, using two namespaces, one is for hbase
>>> cluster, and the other one is for common use.At the same time, we use
>>> Quorum Journal policy as HA.
>>>
>>> GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace
>>>
>>> GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace.
>>>
>>> GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode
>>>
>>> After launching the hdfs cluster all, there is something which makes me
>>> confused, that each namespace has half of the datanodes.
>>>
>>> NameNode 'GS-CIX-SEV0004:9100'
>>>
>>> Started: Thu Jul 04 10:28:00 CST 2013
>>> Version: 2.0.5-alpha, 1488459
>>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>>> Cluster ID: CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7
>>> Block Pool ID: BP-1792015895-10.100.2.3-1372904504940
>>>
>>>
>>> Browse the filesystem
>>> NameNode Logs
>>> Go back to DFS home
>>>
>>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>>> Live Datanodes : 4
>>>
>>>
>>> Last Admin Configured Used Non DFS Remaining
>>> Used Used Remaining Block Block Pool Failed
>>> Node Contact State Capacity (GB) Used (GB)
>>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>>> (GB)
>>> (GB) Used (GB) Blocks
>>> GS-CIX-SEV0001 1 In Service 888.07 0.00 116.04 772.03
>>> 0.00 ┌────────────┐ 86.93 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0002 1 In Service 888.07 0.00 135.50 752.57
>>> 0.00 ┌────────────┐ 84.74 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0005 1 In Service 888.07 0.00 97.61 790.46
>>> 0.00 ┌────────────┐ 89.01 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0006 1 In Service 888.07 0.00 122.30 765.77
>>> 0.00 ┌────────────┐ 86.23 0 0.00 0.00 0
>>>
>>> └────────────┘
>>>
>>>
>>> Another Namespace's NameNode:
>>>
>>> NameNode 'GS-CIX-SEV0001:9100'
>>>
>>> Started: Thu Jul 04 10:19:03 CST 2013
>>> Version: 2.0.5-alpha, 1488459
>>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>>> Cluster ID: CID-1a53483d-000e-4726-aef1-f500bedb1df6
>>> Block Pool ID: BP-1142418822-10.100.2.1-1372904314309
>>>
>>>
>>> Browse the filesystem
>>> NameNode Logs
>>> Go back to DFS home
>>>
>>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>>> Live Datanodes : 4
>>>
>>>
>>> Last Admin Configured Used Non DFS Remaining
>>> Used Used Remaining Block Block Pool Failed
>>> Node Contact State Capacity (GB) Used (GB)
>>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>>> (GB)
>>> (GB) Used (GB) Blocks
>>> GS-CIX-SEV0003 0 In Service 888.07 0.00 150.54 737.53
>>> 0.00 ┌────────────┐ 83.05 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0004 0 In Service 888.07 0.00 177.22 710.85
>>> 0.00 ┌────────────┐ 80.04 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0007 0 In Service 888.07 0.00 62.91 825.16
>>> 0.00 ┌────────────┐ 92.92 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0008 0 In Service 888.07 0.00 125.25 762.82
>>> 0.00 ┌────────────┐ 85.90 0 0.00 0.00 0
>>>
>>>
>>>
>>> And check the DN(GS-CIX-SEV0001)'s log, it prints like this:
>>> 2013-07-04 10:34:51,699 FATAL
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
>>> block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id
>>> DS-1677272131-10.100.2.1-50010-1372905291690) service to GS-CIX-SEV0001/
>>> 10.100.2.1:9100
>>> java.io.IOException: Inconsistent storage IDs. Name-node returned
>>> DS811369792. Expecting DS-1677272131-10.100.2.1-50010-1372905291690
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:731)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:308)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:632)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> It is proved that one datanode has been required to attached to only one
>>> namespace?
>>>
>>> Any views about it will be thankful.
>>>
>>> Regards~
>>>
>>>
>>> --
>>> Bing Jiang
>>> Tel：(86)134-2619-1361
>>> weibo: http://weibo.com/jiangbinglover
>>> BLOG: http://blog.sina.com.cn/jiangbinglover
>>> National Research Center for Intelligent Computing Systems
>>> Institute of Computing technology
>>> Graduate University of Chinese Academy of Science
>>>
>>
>>
>
--
Bing Jiang
Tel：(86)134-2619-1361
weibo: http://weibo.com/jiangbinglover
BLOG: http://blog.sina.com.cn/jiangbinglover
National Research Center for Intelligent Computing Systems
Institute of Computing technology
Graduate University of Chinese Academy of Science
Re: Datanode support different Namespace
Posted by Bing Jiang <ji...@gmail.com>.
If not set cluster id in formatting the Namenode, is there a policy in hdfs
to guarantee the even of distributing DataNodes into different Namespace,
or just randomly?
2013/7/4 Azuryy Yu <az...@gmail.com>
> Additional,
>
> If these are two new clusters, then on each namenode, using "hdfs namenode
> -format -clusterID yourID"
>
> But if you want to upgrade these two clusters from NonHA to HA, then using
> "bin/start-dfs.sh -upgrade -clusterID yourID"
>
>
>
> On Thu, Jul 4, 2013 at 3:14 PM, Azuryy Yu <az...@gmail.com> wrote:
>
>>
>> This is because you don't use the same clusterID. all data nodes and
>> namenodes should use the same clusterID.
>>
>>
>> On Thu, Jul 4, 2013 at 3:12 PM, Bing Jiang <ji...@gmail.com>wrote:
>>
>>> Hi, all
>>>
>>> We try to use hadoop-2.0.5-alpha, using two namespaces, one is for hbase
>>> cluster, and the other one is for common use.At the same time, we use
>>> Quorum Journal policy as HA.
>>>
>>> GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace
>>>
>>> GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace.
>>>
>>> GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode
>>>
>>> After launching the hdfs cluster all, there is something which makes me
>>> confused, that each namespace has half of the datanodes.
>>>
>>> NameNode 'GS-CIX-SEV0004:9100'
>>>
>>> Started: Thu Jul 04 10:28:00 CST 2013
>>> Version: 2.0.5-alpha, 1488459
>>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>>> Cluster ID: CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7
>>> Block Pool ID: BP-1792015895-10.100.2.3-1372904504940
>>>
>>>
>>> Browse the filesystem
>>> NameNode Logs
>>> Go back to DFS home
>>>
>>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>>> Live Datanodes : 4
>>>
>>>
>>> Last Admin Configured Used Non DFS Remaining
>>> Used Used Remaining Block Block Pool Failed
>>> Node Contact State Capacity (GB) Used (GB)
>>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>>> (GB)
>>> (GB) Used (GB) Blocks
>>> GS-CIX-SEV0001 1 In Service 888.07 0.00 116.04 772.03
>>> 0.00 ┌────────────┐ 86.93 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0002 1 In Service 888.07 0.00 135.50 752.57
>>> 0.00 ┌────────────┐ 84.74 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0005 1 In Service 888.07 0.00 97.61 790.46
>>> 0.00 ┌────────────┐ 89.01 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0006 1 In Service 888.07 0.00 122.30 765.77
>>> 0.00 ┌────────────┐ 86.23 0 0.00 0.00 0
>>>
>>> └────────────┘
>>>
>>>
>>> Another Namespace's NameNode:
>>>
>>> NameNode 'GS-CIX-SEV0001:9100'
>>>
>>> Started: Thu Jul 04 10:19:03 CST 2013
>>> Version: 2.0.5-alpha, 1488459
>>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>>> Cluster ID: CID-1a53483d-000e-4726-aef1-f500bedb1df6
>>> Block Pool ID: BP-1142418822-10.100.2.1-1372904314309
>>>
>>>
>>> Browse the filesystem
>>> NameNode Logs
>>> Go back to DFS home
>>>
>>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>>> Live Datanodes : 4
>>>
>>>
>>> Last Admin Configured Used Non DFS Remaining
>>> Used Used Remaining Block Block Pool Failed
>>> Node Contact State Capacity (GB) Used (GB)
>>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>>> (GB)
>>> (GB) Used (GB) Blocks
>>> GS-CIX-SEV0003 0 In Service 888.07 0.00 150.54 737.53
>>> 0.00 ┌────────────┐ 83.05 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0004 0 In Service 888.07 0.00 177.22 710.85
>>> 0.00 ┌────────────┐ 80.04 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0007 0 In Service 888.07 0.00 62.91 825.16
>>> 0.00 ┌────────────┐ 92.92 0 0.00 0.00 0
>>>
>>> └────────────┘
>>> GS-CIX-SEV0008 0 In Service 888.07 0.00 125.25 762.82
>>> 0.00 ┌────────────┐ 85.90 0 0.00 0.00 0
>>>
>>>
>>>
>>> And check the DN(GS-CIX-SEV0001)'s log, it prints like this:
>>> 2013-07-04 10:34:51,699 FATAL
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
>>> block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id
>>> DS-1677272131-10.100.2.1-50010-1372905291690) service to GS-CIX-SEV0001/
>>> 10.100.2.1:9100
>>> java.io.IOException: Inconsistent storage IDs. Name-node returned
>>> DS811369792. Expecting DS-1677272131-10.100.2.1-50010-1372905291690
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:731)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:308)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:632)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> It is proved that one datanode has been required to attached to only one
>>> namespace?
>>>
>>> Any views about it will be thankful.
>>>
>>> Regards~
>>>
>>>
>>> --
>>> Bing Jiang
>>> Tel：(86)134-2619-1361
>>> weibo: http://weibo.com/jiangbinglover
>>> BLOG: http://blog.sina.com.cn/jiangbinglover
>>> National Research Center for Intelligent Computing Systems
>>> Institute of Computing technology
>>> Graduate University of Chinese Academy of Science
>>>
>>
>>
>
--
Bing Jiang
Tel：(86)134-2619-1361
weibo: http://weibo.com/jiangbinglover
BLOG: http://blog.sina.com.cn/jiangbinglover
National Research Center for Intelligent Computing Systems
Institute of Computing technology
Graduate University of Chinese Academy of Science
Re: Datanode support different Namespace
Posted by Azuryy Yu <az...@gmail.com>.
Additional,
If these are two new clusters, then on each namenode, using "hdfs namenode
-format -clusterID yourID"
But if you want to upgrade these two clusters from NonHA to HA, then using
"bin/start-dfs.sh -upgrade -clusterID yourID"
On Thu, Jul 4, 2013 at 3:14 PM, Azuryy Yu <az...@gmail.com> wrote:
>
> This is because you don't use the same clusterID. all data nodes and
> namenodes should use the same clusterID.
>
>
> On Thu, Jul 4, 2013 at 3:12 PM, Bing Jiang <ji...@gmail.com>wrote:
>
>> Hi, all
>>
>> We try to use hadoop-2.0.5-alpha, using two namespaces, one is for hbase
>> cluster, and the other one is for common use.At the same time, we use
>> Quorum Journal policy as HA.
>>
>> GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace
>>
>> GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace.
>>
>> GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode
>>
>> After launching the hdfs cluster all, there is something which makes me
>> confused, that each namespace has half of the datanodes.
>>
>> NameNode 'GS-CIX-SEV0004:9100'
>>
>> Started: Thu Jul 04 10:28:00 CST 2013
>> Version: 2.0.5-alpha, 1488459
>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>> Cluster ID: CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7
>> Block Pool ID: BP-1792015895-10.100.2.3-1372904504940
>>
>>
>> Browse the filesystem
>> NameNode Logs
>> Go back to DFS home
>>
>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>> Live Datanodes : 4
>>
>>
>> Last Admin Configured Used Non DFS Remaining
>> Used Used Remaining Block Block Pool Failed
>> Node Contact State Capacity (GB) Used (GB)
>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>> (GB)
>> (GB) Used (GB) Blocks
>> GS-CIX-SEV0001 1 In Service 888.07 0.00 116.04 772.03
>> 0.00 ┌────────────┐ 86.93 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0002 1 In Service 888.07 0.00 135.50 752.57
>> 0.00 ┌────────────┐ 84.74 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0005 1 In Service 888.07 0.00 97.61 790.46
>> 0.00 ┌────────────┐ 89.01 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0006 1 In Service 888.07 0.00 122.30 765.77
>> 0.00 ┌────────────┐ 86.23 0 0.00 0.00 0
>>
>> └────────────┘
>>
>>
>> Another Namespace's NameNode:
>>
>> NameNode 'GS-CIX-SEV0001:9100'
>>
>> Started: Thu Jul 04 10:19:03 CST 2013
>> Version: 2.0.5-alpha, 1488459
>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>> Cluster ID: CID-1a53483d-000e-4726-aef1-f500bedb1df6
>> Block Pool ID: BP-1142418822-10.100.2.1-1372904314309
>>
>>
>> Browse the filesystem
>> NameNode Logs
>> Go back to DFS home
>>
>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>> Live Datanodes : 4
>>
>>
>> Last Admin Configured Used Non DFS Remaining
>> Used Used Remaining Block Block Pool Failed
>> Node Contact State Capacity (GB) Used (GB)
>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>> (GB)
>> (GB) Used (GB) Blocks
>> GS-CIX-SEV0003 0 In Service 888.07 0.00 150.54 737.53
>> 0.00 ┌────────────┐ 83.05 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0004 0 In Service 888.07 0.00 177.22 710.85
>> 0.00 ┌────────────┐ 80.04 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0007 0 In Service 888.07 0.00 62.91 825.16
>> 0.00 ┌────────────┐ 92.92 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0008 0 In Service 888.07 0.00 125.25 762.82
>> 0.00 ┌────────────┐ 85.90 0 0.00 0.00 0
>>
>>
>>
>> And check the DN(GS-CIX-SEV0001)'s log, it prints like this:
>> 2013-07-04 10:34:51,699 FATAL
>> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
>> block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id
>> DS-1677272131-10.100.2.1-50010-1372905291690) service to GS-CIX-SEV0001/
>> 10.100.2.1:9100
>> java.io.IOException: Inconsistent storage IDs. Name-node returned
>> DS811369792. Expecting DS-1677272131-10.100.2.1-50010-1372905291690
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:731)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:308)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:632)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> It is proved that one datanode has been required to attached to only one
>> namespace?
>>
>> Any views about it will be thankful.
>>
>> Regards~
>>
>>
>> --
>> Bing Jiang
>> Tel：(86)134-2619-1361
>> weibo: http://weibo.com/jiangbinglover
>> BLOG: http://blog.sina.com.cn/jiangbinglover
>> National Research Center for Intelligent Computing Systems
>> Institute of Computing technology
>> Graduate University of Chinese Academy of Science
>>
>
>
Re: Datanode support different Namespace
Posted by Azuryy Yu <az...@gmail.com>.
Additional,
If these are two new clusters, then on each namenode, using "hdfs namenode
-format -clusterID yourID"
But if you want to upgrade these two clusters from NonHA to HA, then using
"bin/start-dfs.sh -upgrade -clusterID yourID"
On Thu, Jul 4, 2013 at 3:14 PM, Azuryy Yu <az...@gmail.com> wrote:
>
> This is because you don't use the same clusterID. all data nodes and
> namenodes should use the same clusterID.
>
>
> On Thu, Jul 4, 2013 at 3:12 PM, Bing Jiang <ji...@gmail.com>wrote:
>
>> Hi, all
>>
>> We try to use hadoop-2.0.5-alpha, using two namespaces, one is for hbase
>> cluster, and the other one is for common use.At the same time, we use
>> Quorum Journal policy as HA.
>>
>> GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace
>>
>> GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace.
>>
>> GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode
>>
>> After launching the hdfs cluster all, there is something which makes me
>> confused, that each namespace has half of the datanodes.
>>
>> NameNode 'GS-CIX-SEV0004:9100'
>>
>> Started: Thu Jul 04 10:28:00 CST 2013
>> Version: 2.0.5-alpha, 1488459
>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>> Cluster ID: CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7
>> Block Pool ID: BP-1792015895-10.100.2.3-1372904504940
>>
>>
>> Browse the filesystem
>> NameNode Logs
>> Go back to DFS home
>>
>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>> Live Datanodes : 4
>>
>>
>> Last Admin Configured Used Non DFS Remaining
>> Used Used Remaining Block Block Pool Failed
>> Node Contact State Capacity (GB) Used (GB)
>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>> (GB)
>> (GB) Used (GB) Blocks
>> GS-CIX-SEV0001 1 In Service 888.07 0.00 116.04 772.03
>> 0.00 ┌────────────┐ 86.93 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0002 1 In Service 888.07 0.00 135.50 752.57
>> 0.00 ┌────────────┐ 84.74 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0005 1 In Service 888.07 0.00 97.61 790.46
>> 0.00 ┌────────────┐ 89.01 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0006 1 In Service 888.07 0.00 122.30 765.77
>> 0.00 ┌────────────┐ 86.23 0 0.00 0.00 0
>>
>> └────────────┘
>>
>>
>> Another Namespace's NameNode:
>>
>> NameNode 'GS-CIX-SEV0001:9100'
>>
>> Started: Thu Jul 04 10:19:03 CST 2013
>> Version: 2.0.5-alpha, 1488459
>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>> Cluster ID: CID-1a53483d-000e-4726-aef1-f500bedb1df6
>> Block Pool ID: BP-1142418822-10.100.2.1-1372904314309
>>
>>
>> Browse the filesystem
>> NameNode Logs
>> Go back to DFS home
>>
>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>> Live Datanodes : 4
>>
>>
>> Last Admin Configured Used Non DFS Remaining
>> Used Used Remaining Block Block Pool Failed
>> Node Contact State Capacity (GB) Used (GB)
>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>> (GB)
>> (GB) Used (GB) Blocks
>> GS-CIX-SEV0003 0 In Service 888.07 0.00 150.54 737.53
>> 0.00 ┌────────────┐ 83.05 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0004 0 In Service 888.07 0.00 177.22 710.85
>> 0.00 ┌────────────┐ 80.04 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0007 0 In Service 888.07 0.00 62.91 825.16
>> 0.00 ┌────────────┐ 92.92 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0008 0 In Service 888.07 0.00 125.25 762.82
>> 0.00 ┌────────────┐ 85.90 0 0.00 0.00 0
>>
>>
>>
>> And check the DN(GS-CIX-SEV0001)'s log, it prints like this:
>> 2013-07-04 10:34:51,699 FATAL
>> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
>> block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id
>> DS-1677272131-10.100.2.1-50010-1372905291690) service to GS-CIX-SEV0001/
>> 10.100.2.1:9100
>> java.io.IOException: Inconsistent storage IDs. Name-node returned
>> DS811369792. Expecting DS-1677272131-10.100.2.1-50010-1372905291690
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:731)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:308)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:632)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> It is proved that one datanode has been required to attached to only one
>> namespace?
>>
>> Any views about it will be thankful.
>>
>> Regards~
>>
>>
>> --
>> Bing Jiang
>> Tel：(86)134-2619-1361
>> weibo: http://weibo.com/jiangbinglover
>> BLOG: http://blog.sina.com.cn/jiangbinglover
>> National Research Center for Intelligent Computing Systems
>> Institute of Computing technology
>> Graduate University of Chinese Academy of Science
>>
>
>
Re: Datanode support different Namespace
Posted by Azuryy Yu <az...@gmail.com>.
Additional,
If these are two new clusters, then on each namenode, using "hdfs namenode
-format -clusterID yourID"
But if you want to upgrade these two clusters from NonHA to HA, then using
"bin/start-dfs.sh -upgrade -clusterID yourID"
On Thu, Jul 4, 2013 at 3:14 PM, Azuryy Yu <az...@gmail.com> wrote:
>
> This is because you don't use the same clusterID. all data nodes and
> namenodes should use the same clusterID.
>
>
> On Thu, Jul 4, 2013 at 3:12 PM, Bing Jiang <ji...@gmail.com>wrote:
>
>> Hi, all
>>
>> We try to use hadoop-2.0.5-alpha, using two namespaces, one is for hbase
>> cluster, and the other one is for common use.At the same time, we use
>> Quorum Journal policy as HA.
>>
>> GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace
>>
>> GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace.
>>
>> GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode
>>
>> After launching the hdfs cluster all, there is something which makes me
>> confused, that each namespace has half of the datanodes.
>>
>> NameNode 'GS-CIX-SEV0004:9100'
>>
>> Started: Thu Jul 04 10:28:00 CST 2013
>> Version: 2.0.5-alpha, 1488459
>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>> Cluster ID: CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7
>> Block Pool ID: BP-1792015895-10.100.2.3-1372904504940
>>
>>
>> Browse the filesystem
>> NameNode Logs
>> Go back to DFS home
>>
>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>> Live Datanodes : 4
>>
>>
>> Last Admin Configured Used Non DFS Remaining
>> Used Used Remaining Block Block Pool Failed
>> Node Contact State Capacity (GB) Used (GB)
>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>> (GB)
>> (GB) Used (GB) Blocks
>> GS-CIX-SEV0001 1 In Service 888.07 0.00 116.04 772.03
>> 0.00 ┌────────────┐ 86.93 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0002 1 In Service 888.07 0.00 135.50 752.57
>> 0.00 ┌────────────┐ 84.74 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0005 1 In Service 888.07 0.00 97.61 790.46
>> 0.00 ┌────────────┐ 89.01 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0006 1 In Service 888.07 0.00 122.30 765.77
>> 0.00 ┌────────────┐ 86.23 0 0.00 0.00 0
>>
>> └────────────┘
>>
>>
>> Another Namespace's NameNode:
>>
>> NameNode 'GS-CIX-SEV0001:9100'
>>
>> Started: Thu Jul 04 10:19:03 CST 2013
>> Version: 2.0.5-alpha, 1488459
>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>> Cluster ID: CID-1a53483d-000e-4726-aef1-f500bedb1df6
>> Block Pool ID: BP-1142418822-10.100.2.1-1372904314309
>>
>>
>> Browse the filesystem
>> NameNode Logs
>> Go back to DFS home
>>
>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>> Live Datanodes : 4
>>
>>
>> Last Admin Configured Used Non DFS Remaining
>> Used Used Remaining Block Block Pool Failed
>> Node Contact State Capacity (GB) Used (GB)
>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>> (GB)
>> (GB) Used (GB) Blocks
>> GS-CIX-SEV0003 0 In Service 888.07 0.00 150.54 737.53
>> 0.00 ┌────────────┐ 83.05 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0004 0 In Service 888.07 0.00 177.22 710.85
>> 0.00 ┌────────────┐ 80.04 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0007 0 In Service 888.07 0.00 62.91 825.16
>> 0.00 ┌────────────┐ 92.92 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0008 0 In Service 888.07 0.00 125.25 762.82
>> 0.00 ┌────────────┐ 85.90 0 0.00 0.00 0
>>
>>
>>
>> And check the DN(GS-CIX-SEV0001)'s log, it prints like this:
>> 2013-07-04 10:34:51,699 FATAL
>> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
>> block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id
>> DS-1677272131-10.100.2.1-50010-1372905291690) service to GS-CIX-SEV0001/
>> 10.100.2.1:9100
>> java.io.IOException: Inconsistent storage IDs. Name-node returned
>> DS811369792. Expecting DS-1677272131-10.100.2.1-50010-1372905291690
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:731)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:308)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:632)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> It is proved that one datanode has been required to attached to only one
>> namespace?
>>
>> Any views about it will be thankful.
>>
>> Regards~
>>
>>
>> --
>> Bing Jiang
>> Tel：(86)134-2619-1361
>> weibo: http://weibo.com/jiangbinglover
>> BLOG: http://blog.sina.com.cn/jiangbinglover
>> National Research Center for Intelligent Computing Systems
>> Institute of Computing technology
>> Graduate University of Chinese Academy of Science
>>
>
>
Re: Datanode support different Namespace
Posted by Azuryy Yu <az...@gmail.com>.
Additional,
If these are two new clusters, then on each namenode, using "hdfs namenode
-format -clusterID yourID"
But if you want to upgrade these two clusters from NonHA to HA, then using
"bin/start-dfs.sh -upgrade -clusterID yourID"
On Thu, Jul 4, 2013 at 3:14 PM, Azuryy Yu <az...@gmail.com> wrote:
>
> This is because you don't use the same clusterID. all data nodes and
> namenodes should use the same clusterID.
>
>
> On Thu, Jul 4, 2013 at 3:12 PM, Bing Jiang <ji...@gmail.com>wrote:
>
>> Hi, all
>>
>> We try to use hadoop-2.0.5-alpha, using two namespaces, one is for hbase
>> cluster, and the other one is for common use.At the same time, we use
>> Quorum Journal policy as HA.
>>
>> GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace
>>
>> GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace.
>>
>> GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode
>>
>> After launching the hdfs cluster all, there is something which makes me
>> confused, that each namespace has half of the datanodes.
>>
>> NameNode 'GS-CIX-SEV0004:9100'
>>
>> Started: Thu Jul 04 10:28:00 CST 2013
>> Version: 2.0.5-alpha, 1488459
>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>> Cluster ID: CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7
>> Block Pool ID: BP-1792015895-10.100.2.3-1372904504940
>>
>>
>> Browse the filesystem
>> NameNode Logs
>> Go back to DFS home
>>
>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>> Live Datanodes : 4
>>
>>
>> Last Admin Configured Used Non DFS Remaining
>> Used Used Remaining Block Block Pool Failed
>> Node Contact State Capacity (GB) Used (GB)
>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>> (GB)
>> (GB) Used (GB) Blocks
>> GS-CIX-SEV0001 1 In Service 888.07 0.00 116.04 772.03
>> 0.00 ┌────────────┐ 86.93 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0002 1 In Service 888.07 0.00 135.50 752.57
>> 0.00 ┌────────────┐ 84.74 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0005 1 In Service 888.07 0.00 97.61 790.46
>> 0.00 ┌────────────┐ 89.01 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0006 1 In Service 888.07 0.00 122.30 765.77
>> 0.00 ┌────────────┐ 86.23 0 0.00 0.00 0
>>
>> └────────────┘
>>
>>
>> Another Namespace's NameNode:
>>
>> NameNode 'GS-CIX-SEV0001:9100'
>>
>> Started: Thu Jul 04 10:19:03 CST 2013
>> Version: 2.0.5-alpha, 1488459
>> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
>> Cluster ID: CID-1a53483d-000e-4726-aef1-f500bedb1df6
>> Block Pool ID: BP-1142418822-10.100.2.1-1372904314309
>>
>>
>> Browse the filesystem
>> NameNode Logs
>> Go back to DFS home
>>
>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>> Live Datanodes : 4
>>
>>
>> Last Admin Configured Used Non DFS Remaining
>> Used Used Remaining Block Block Pool Failed
>> Node Contact State Capacity (GB) Used (GB)
>> (%) (%) (%) Blocks Pool Used (%)> Volumes
>> (GB)
>> (GB) Used (GB) Blocks
>> GS-CIX-SEV0003 0 In Service 888.07 0.00 150.54 737.53
>> 0.00 ┌────────────┐ 83.05 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0004 0 In Service 888.07 0.00 177.22 710.85
>> 0.00 ┌────────────┐ 80.04 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0007 0 In Service 888.07 0.00 62.91 825.16
>> 0.00 ┌────────────┐ 92.92 0 0.00 0.00 0
>>
>> └────────────┘
>> GS-CIX-SEV0008 0 In Service 888.07 0.00 125.25 762.82
>> 0.00 ┌────────────┐ 85.90 0 0.00 0.00 0
>>
>>
>>
>> And check the DN(GS-CIX-SEV0001)'s log, it prints like this:
>> 2013-07-04 10:34:51,699 FATAL
>> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
>> block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id
>> DS-1677272131-10.100.2.1-50010-1372905291690) service to GS-CIX-SEV0001/
>> 10.100.2.1:9100
>> java.io.IOException: Inconsistent storage IDs. Name-node returned
>> DS811369792. Expecting DS-1677272131-10.100.2.1-50010-1372905291690
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:731)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:308)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:632)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> It is proved that one datanode has been required to attached to only one
>> namespace?
>>
>> Any views about it will be thankful.
>>
>> Regards~
>>
>>
>> --
>> Bing Jiang
>> Tel：(86)134-2619-1361
>> weibo: http://weibo.com/jiangbinglover
>> BLOG: http://blog.sina.com.cn/jiangbinglover
>> National Research Center for Intelligent Computing Systems
>> Institute of Computing technology
>> Graduate University of Chinese Academy of Science
>>
>
>
Re: Datanode support different Namespace
Posted by Azuryy Yu <az...@gmail.com>.
This is because you don't use the same clusterID. all data nodes and
namenodes should use the same clusterID.
On Thu, Jul 4, 2013 at 3:12 PM, Bing Jiang <ji...@gmail.com> wrote:
> Hi, all
>
> We try to use hadoop-2.0.5-alpha, using two namespaces, one is for hbase
> cluster, and the other one is for common use.At the same time, we use
> Quorum Journal policy as HA.
>
> GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace
>
> GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace.
>
> GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode
>
> After launching the hdfs cluster all, there is something which makes me
> confused, that each namespace has half of the datanodes.
>
> NameNode 'GS-CIX-SEV0004:9100'
>
> Started: Thu Jul 04 10:28:00 CST 2013
> Version: 2.0.5-alpha, 1488459
> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
> Cluster ID: CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7
> Block Pool ID: BP-1792015895-10.100.2.3-1372904504940
>
>
> Browse the filesystem
> NameNode Logs
> Go back to DFS home
>
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
> Live Datanodes : 4
>
>
> Last Admin Configured Used Non DFS Remaining
> Used Used Remaining Block Block Pool Failed
> Node Contact State Capacity (GB) Used (GB)
> (%) (%) (%) Blocks Pool Used (%)> Volumes
> (GB)
> (GB) Used (GB) Blocks
> GS-CIX-SEV0001 1 In Service 888.07 0.00 116.04 772.03 0.00
> ┌────────────┐ 86.93 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0002 1 In Service 888.07 0.00 135.50 752.57 0.00
> ┌────────────┐ 84.74 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0005 1 In Service 888.07 0.00 97.61 790.46 0.00
> ┌────────────┐ 89.01 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0006 1 In Service 888.07 0.00 122.30 765.77 0.00
> ┌────────────┐ 86.23 0 0.00 0.00 0
>
> └────────────┘
>
>
> Another Namespace's NameNode:
>
> NameNode 'GS-CIX-SEV0001:9100'
>
> Started: Thu Jul 04 10:19:03 CST 2013
> Version: 2.0.5-alpha, 1488459
> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
> Cluster ID: CID-1a53483d-000e-4726-aef1-f500bedb1df6
> Block Pool ID: BP-1142418822-10.100.2.1-1372904314309
>
>
> Browse the filesystem
> NameNode Logs
> Go back to DFS home
>
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
> Live Datanodes : 4
>
>
> Last Admin Configured Used Non DFS Remaining
> Used Used Remaining Block Block Pool Failed
> Node Contact State Capacity (GB) Used (GB)
> (%) (%) (%) Blocks Pool Used (%)> Volumes
> (GB)
> (GB) Used (GB) Blocks
> GS-CIX-SEV0003 0 In Service 888.07 0.00 150.54 737.53 0.00
> ┌────────────┐ 83.05 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0004 0 In Service 888.07 0.00 177.22 710.85 0.00
> ┌────────────┐ 80.04 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0007 0 In Service 888.07 0.00 62.91 825.16 0.00
> ┌────────────┐ 92.92 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0008 0 In Service 888.07 0.00 125.25 762.82 0.00
> ┌────────────┐ 85.90 0 0.00 0.00 0
>
>
>
> And check the DN(GS-CIX-SEV0001)'s log, it prints like this:
> 2013-07-04 10:34:51,699 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
> block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id
> DS-1677272131-10.100.2.1-50010-1372905291690) service to GS-CIX-SEV0001/
> 10.100.2.1:9100
> java.io.IOException: Inconsistent storage IDs. Name-node returned
> DS811369792. Expecting DS-1677272131-10.100.2.1-50010-1372905291690
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:731)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:308)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:632)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
> at java.lang.Thread.run(Thread.java:662)
>
> It is proved that one datanode has been required to attached to only one
> namespace?
>
> Any views about it will be thankful.
>
> Regards~
>
>
> --
> Bing Jiang
> Tel：(86)134-2619-1361
> weibo: http://weibo.com/jiangbinglover
> BLOG: http://blog.sina.com.cn/jiangbinglover
> National Research Center for Intelligent Computing Systems
> Institute of Computing technology
> Graduate University of Chinese Academy of Science
>
Re: Datanode support different Namespace
Posted by Azuryy Yu <az...@gmail.com>.
This is because you don't use the same clusterID. all data nodes and
namenodes should use the same clusterID.
On Thu, Jul 4, 2013 at 3:12 PM, Bing Jiang <ji...@gmail.com> wrote:
> Hi, all
>
> We try to use hadoop-2.0.5-alpha, using two namespaces, one is for hbase
> cluster, and the other one is for common use.At the same time, we use
> Quorum Journal policy as HA.
>
> GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace
>
> GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace.
>
> GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode
>
> After launching the hdfs cluster all, there is something which makes me
> confused, that each namespace has half of the datanodes.
>
> NameNode 'GS-CIX-SEV0004:9100'
>
> Started: Thu Jul 04 10:28:00 CST 2013
> Version: 2.0.5-alpha, 1488459
> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
> Cluster ID: CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7
> Block Pool ID: BP-1792015895-10.100.2.3-1372904504940
>
>
> Browse the filesystem
> NameNode Logs
> Go back to DFS home
>
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
> Live Datanodes : 4
>
>
> Last Admin Configured Used Non DFS Remaining
> Used Used Remaining Block Block Pool Failed
> Node Contact State Capacity (GB) Used (GB)
> (%) (%) (%) Blocks Pool Used (%)> Volumes
> (GB)
> (GB) Used (GB) Blocks
> GS-CIX-SEV0001 1 In Service 888.07 0.00 116.04 772.03 0.00
> ┌────────────┐ 86.93 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0002 1 In Service 888.07 0.00 135.50 752.57 0.00
> ┌────────────┐ 84.74 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0005 1 In Service 888.07 0.00 97.61 790.46 0.00
> ┌────────────┐ 89.01 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0006 1 In Service 888.07 0.00 122.30 765.77 0.00
> ┌────────────┐ 86.23 0 0.00 0.00 0
>
> └────────────┘
>
>
> Another Namespace's NameNode:
>
> NameNode 'GS-CIX-SEV0001:9100'
>
> Started: Thu Jul 04 10:19:03 CST 2013
> Version: 2.0.5-alpha, 1488459
> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
> Cluster ID: CID-1a53483d-000e-4726-aef1-f500bedb1df6
> Block Pool ID: BP-1142418822-10.100.2.1-1372904314309
>
>
> Browse the filesystem
> NameNode Logs
> Go back to DFS home
>
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
> Live Datanodes : 4
>
>
> Last Admin Configured Used Non DFS Remaining
> Used Used Remaining Block Block Pool Failed
> Node Contact State Capacity (GB) Used (GB)
> (%) (%) (%) Blocks Pool Used (%)> Volumes
> (GB)
> (GB) Used (GB) Blocks
> GS-CIX-SEV0003 0 In Service 888.07 0.00 150.54 737.53 0.00
> ┌────────────┐ 83.05 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0004 0 In Service 888.07 0.00 177.22 710.85 0.00
> ┌────────────┐ 80.04 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0007 0 In Service 888.07 0.00 62.91 825.16 0.00
> ┌────────────┐ 92.92 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0008 0 In Service 888.07 0.00 125.25 762.82 0.00
> ┌────────────┐ 85.90 0 0.00 0.00 0
>
>
>
> And check the DN(GS-CIX-SEV0001)'s log, it prints like this:
> 2013-07-04 10:34:51,699 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
> block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id
> DS-1677272131-10.100.2.1-50010-1372905291690) service to GS-CIX-SEV0001/
> 10.100.2.1:9100
> java.io.IOException: Inconsistent storage IDs. Name-node returned
> DS811369792. Expecting DS-1677272131-10.100.2.1-50010-1372905291690
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:731)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:308)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:632)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
> at java.lang.Thread.run(Thread.java:662)
>
> It is proved that one datanode has been required to attached to only one
> namespace?
>
> Any views about it will be thankful.
>
> Regards~
>
>
> --
> Bing Jiang
> Tel：(86)134-2619-1361
> weibo: http://weibo.com/jiangbinglover
> BLOG: http://blog.sina.com.cn/jiangbinglover
> National Research Center for Intelligent Computing Systems
> Institute of Computing technology
> Graduate University of Chinese Academy of Science
>
Re: Datanode support different Namespace
Posted by Azuryy Yu <az...@gmail.com>.
This is because you don't use the same clusterID. all data nodes and
namenodes should use the same clusterID.
On Thu, Jul 4, 2013 at 3:12 PM, Bing Jiang <ji...@gmail.com> wrote:
> Hi, all
>
> We try to use hadoop-2.0.5-alpha, using two namespaces, one is for hbase
> cluster, and the other one is for common use.At the same time, we use
> Quorum Journal policy as HA.
>
> GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace
>
> GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace.
>
> GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode
>
> After launching the hdfs cluster all, there is something which makes me
> confused, that each namespace has half of the datanodes.
>
> NameNode 'GS-CIX-SEV0004:9100'
>
> Started: Thu Jul 04 10:28:00 CST 2013
> Version: 2.0.5-alpha, 1488459
> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
> Cluster ID: CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7
> Block Pool ID: BP-1792015895-10.100.2.3-1372904504940
>
>
> Browse the filesystem
> NameNode Logs
> Go back to DFS home
>
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
> Live Datanodes : 4
>
>
> Last Admin Configured Used Non DFS Remaining
> Used Used Remaining Block Block Pool Failed
> Node Contact State Capacity (GB) Used (GB)
> (%) (%) (%) Blocks Pool Used (%)> Volumes
> (GB)
> (GB) Used (GB) Blocks
> GS-CIX-SEV0001 1 In Service 888.07 0.00 116.04 772.03 0.00
> ┌────────────┐ 86.93 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0002 1 In Service 888.07 0.00 135.50 752.57 0.00
> ┌────────────┐ 84.74 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0005 1 In Service 888.07 0.00 97.61 790.46 0.00
> ┌────────────┐ 89.01 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0006 1 In Service 888.07 0.00 122.30 765.77 0.00
> ┌────────────┐ 86.23 0 0.00 0.00 0
>
> └────────────┘
>
>
> Another Namespace's NameNode:
>
> NameNode 'GS-CIX-SEV0001:9100'
>
> Started: Thu Jul 04 10:19:03 CST 2013
> Version: 2.0.5-alpha, 1488459
> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
> Cluster ID: CID-1a53483d-000e-4726-aef1-f500bedb1df6
> Block Pool ID: BP-1142418822-10.100.2.1-1372904314309
>
>
> Browse the filesystem
> NameNode Logs
> Go back to DFS home
>
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
> Live Datanodes : 4
>
>
> Last Admin Configured Used Non DFS Remaining
> Used Used Remaining Block Block Pool Failed
> Node Contact State Capacity (GB) Used (GB)
> (%) (%) (%) Blocks Pool Used (%)> Volumes
> (GB)
> (GB) Used (GB) Blocks
> GS-CIX-SEV0003 0 In Service 888.07 0.00 150.54 737.53 0.00
> ┌────────────┐ 83.05 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0004 0 In Service 888.07 0.00 177.22 710.85 0.00
> ┌────────────┐ 80.04 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0007 0 In Service 888.07 0.00 62.91 825.16 0.00
> ┌────────────┐ 92.92 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0008 0 In Service 888.07 0.00 125.25 762.82 0.00
> ┌────────────┐ 85.90 0 0.00 0.00 0
>
>
>
> And check the DN(GS-CIX-SEV0001)'s log, it prints like this:
> 2013-07-04 10:34:51,699 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
> block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id
> DS-1677272131-10.100.2.1-50010-1372905291690) service to GS-CIX-SEV0001/
> 10.100.2.1:9100
> java.io.IOException: Inconsistent storage IDs. Name-node returned
> DS811369792. Expecting DS-1677272131-10.100.2.1-50010-1372905291690
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:731)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:308)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:632)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
> at java.lang.Thread.run(Thread.java:662)
>
> It is proved that one datanode has been required to attached to only one
> namespace?
>
> Any views about it will be thankful.
>
> Regards~
>
>
> --
> Bing Jiang
> Tel：(86)134-2619-1361
> weibo: http://weibo.com/jiangbinglover
> BLOG: http://blog.sina.com.cn/jiangbinglover
> National Research Center for Intelligent Computing Systems
> Institute of Computing technology
> Graduate University of Chinese Academy of Science
>
Re: Datanode support different Namespace
Posted by Azuryy Yu <az...@gmail.com>.
This is because you don't use the same clusterID. all data nodes and
namenodes should use the same clusterID.
On Thu, Jul 4, 2013 at 3:12 PM, Bing Jiang <ji...@gmail.com> wrote:
> Hi, all
>
> We try to use hadoop-2.0.5-alpha, using two namespaces, one is for hbase
> cluster, and the other one is for common use.At the same time, we use
> Quorum Journal policy as HA.
>
> GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace
>
> GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace.
>
> GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode
>
> After launching the hdfs cluster all, there is something which makes me
> confused, that each namespace has half of the datanodes.
>
> NameNode 'GS-CIX-SEV0004:9100'
>
> Started: Thu Jul 04 10:28:00 CST 2013
> Version: 2.0.5-alpha, 1488459
> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
> Cluster ID: CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7
> Block Pool ID: BP-1792015895-10.100.2.3-1372904504940
>
>
> Browse the filesystem
> NameNode Logs
> Go back to DFS home
>
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
> Live Datanodes : 4
>
>
> Last Admin Configured Used Non DFS Remaining
> Used Used Remaining Block Block Pool Failed
> Node Contact State Capacity (GB) Used (GB)
> (%) (%) (%) Blocks Pool Used (%)> Volumes
> (GB)
> (GB) Used (GB) Blocks
> GS-CIX-SEV0001 1 In Service 888.07 0.00 116.04 772.03 0.00
> ┌────────────┐ 86.93 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0002 1 In Service 888.07 0.00 135.50 752.57 0.00
> ┌────────────┐ 84.74 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0005 1 In Service 888.07 0.00 97.61 790.46 0.00
> ┌────────────┐ 89.01 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0006 1 In Service 888.07 0.00 122.30 765.77 0.00
> ┌────────────┐ 86.23 0 0.00 0.00 0
>
> └────────────┘
>
>
> Another Namespace's NameNode:
>
> NameNode 'GS-CIX-SEV0001:9100'
>
> Started: Thu Jul 04 10:19:03 CST 2013
> Version: 2.0.5-alpha, 1488459
> Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
> Cluster ID: CID-1a53483d-000e-4726-aef1-f500bedb1df6
> Block Pool ID: BP-1142418822-10.100.2.1-1372904314309
>
>
> Browse the filesystem
> NameNode Logs
> Go back to DFS home
>
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
> Live Datanodes : 4
>
>
> Last Admin Configured Used Non DFS Remaining
> Used Used Remaining Block Block Pool Failed
> Node Contact State Capacity (GB) Used (GB)
> (%) (%) (%) Blocks Pool Used (%)> Volumes
> (GB)
> (GB) Used (GB) Blocks
> GS-CIX-SEV0003 0 In Service 888.07 0.00 150.54 737.53 0.00
> ┌────────────┐ 83.05 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0004 0 In Service 888.07 0.00 177.22 710.85 0.00
> ┌────────────┐ 80.04 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0007 0 In Service 888.07 0.00 62.91 825.16 0.00
> ┌────────────┐ 92.92 0 0.00 0.00 0
>
> └────────────┘
> GS-CIX-SEV0008 0 In Service 888.07 0.00 125.25 762.82 0.00
> ┌────────────┐ 85.90 0 0.00 0.00 0
>
>
>
> And check the DN(GS-CIX-SEV0001)'s log, it prints like this:
> 2013-07-04 10:34:51,699 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
> block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id
> DS-1677272131-10.100.2.1-50010-1372905291690) service to GS-CIX-SEV0001/
> 10.100.2.1:9100
> java.io.IOException: Inconsistent storage IDs. Name-node returned
> DS811369792. Expecting DS-1677272131-10.100.2.1-50010-1372905291690
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:731)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:308)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:632)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
> at java.lang.Thread.run(Thread.java:662)
>
> It is proved that one datanode has been required to attached to only one
> namespace?
>
> Any views about it will be thankful.
>
> Regards~
>
>
> --
> Bing Jiang
> Tel：(86)134-2619-1361
> weibo: http://weibo.com/jiangbinglover
> BLOG: http://blog.sina.com.cn/jiangbinglover
> National Research Center for Intelligent Computing Systems
> Institute of Computing technology
> Graduate University of Chinese Academy of Science
>