You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Gaojinchao <ga...@huawei.com> on 2011/10/11 12:21:05 UTC
Created a invalid Zk node
The below logs said that we created a invalid zk node when restarted a cluster.
We mistakenly believed that the regions belong to a dead server.
Can I open a issue ?
2011-10-11 05:05:29,127 INFO org.apache.hadoop.hbase.master.HMaster: Meta updated status = true
2011-10-11 05:05:29,127 INFO org.apache.hadoop.hbase.master.HMaster: ROOT/Meta already up-to date with new HRI.
2011-10-11 05:05:29,151 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for 771d63e9327383159553619a4f2dc74f with OFFLINE state
2011-10-11 05:05:29,161 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for 3cf860dd323fe6360f571aeafc129f95 with OFFLINE state
2011-10-11 05:05:29,170 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for 4065350214452a9d5c55243c734bef08 with OFFLINE state
2011-10-11 05:05:29,178 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for 4e81613f82a39fc6e5e89f96e7b3ccc4 with OFFLINE state
2011-10-11 05:05:29,187 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for e21b9e1545a28953aba0098fda5c9cd9 with OFFLINE state
2011-10-11 05:05:29,195 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for 5cd9f55eecd43d088bbd505f6795131f with OFFLINE state
2011-10-11 05:05:29,229 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for db5f641452a70b09b85a92970e4198c7 with OFFLINE state
2011-10-11 05:05:29,237 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for a7b20a653919e7f41bfb2ed349af7d21 with OFFLINE state
2011-10-11 05:05:29,253 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for c9385619425f737eab1a6624d2e097a8 with OFFLINE state
// we cleaned the all the zk nodes.
2011-10-11 05:05:29,262 INFO org.apache.hadoop.hbase.master.AssignmentManager: Clean cluster startup. Assigning userregions
2011-10-11 05:05:29,262 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Deleting any existing unassigned nodes
2011-10-11 05:05:29,367 INFO org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 9 region(s) across 1 server(s), retainAssignment=true
2011-10-11 05:05:29,369 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Timeout-on-RIT=9000
2011-10-11 05:05:29,369 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 9 region(s) to C3S3,54366,1318323920153
2011-10-11 05:05:29,369 INFO org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done
2011-10-11 05:05:29,371 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Async create of unassigned node for 771d63e9327383159553619a4f2dc74f with OFFLINE state
2011-10-11 05:05:29,371 INFO org.apache.hadoop.hbase.master.HMaster: Master has completed initialization
2011-10-11 05:05:29,371 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Async create of unassigned node for 3cf860dd323fe6360f571aeafc129f95 with OFFLINE state
2011-10-11 05:05:29,371 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Async create of unassigned node for 4065350214452a9d5c55243c734bef08 with OFFLINE state
2011-10-11 05:05:29,371 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Async create of unassigned node for 4e81613f82a39fc6e5e89f96e7b3ccc4 with OFFLINE state
2011-10-11 05:05:29,371 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Async create of unassigned node for e21b9e1545a28953aba0098fda5c9cd9 with OFFLINE state
2011-10-11 05:05:29,372 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Async create of unassigned node for 5cd9f55eecd43d088bbd505f6795131f with OFFLINE state
2011-10-11 05:05:29,372 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Async create of unassigned node for db5f641452a70b09b85a92970e4198c7 with OFFLINE state
2011-10-11 05:05:29,372 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Async create of unassigned node for a7b20a653919e7f41bfb2ed349af7d21 with OFFLINE state
2011-10-11 05:05:29,372 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Async create of unassigned node for c9385619425f737eab1a6624d2e097a8 with OFFLINE state
RE: Created a invalid Zk node
Posted by Ramkrishna S Vasudevan <ra...@huawei.com>.
Hi
Yes, because we think the server is a dead server as per the current code
flow in trunk, we create znodes in OFFLINE state. Then we again infer that
there were no live region servers and again call assignAllUserRegions() as
part of which we delete the created znodes. Unnecessary creation and
deletion of znodes.
Regards
Ram
-----Original Message-----
From: Gaojinchao [mailto:gaojinchao@huawei.com]
Sent: Tuesday, October 11, 2011 3:51 PM
To: dev@hbase.apache.org
Cc: Chenjian
Subject: Created a invalid Zk node
The below logs said that we created a invalid zk node when restarted a
cluster.
We mistakenly believed that the regions belong to a dead server.
Can I open a issue ?
2011-10-11 05:05:29,127 INFO org.apache.hadoop.hbase.master.HMaster: Meta
updated status = true
2011-10-11 05:05:29,127 INFO org.apache.hadoop.hbase.master.HMaster:
ROOT/Meta already up-to date with new HRI.
2011-10-11 05:05:29,151 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for
771d63e9327383159553619a4f2dc74f with OFFLINE state
2011-10-11 05:05:29,161 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for
3cf860dd323fe6360f571aeafc129f95 with OFFLINE state
2011-10-11 05:05:29,170 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for
4065350214452a9d5c55243c734bef08 with OFFLINE state
2011-10-11 05:05:29,178 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for
4e81613f82a39fc6e5e89f96e7b3ccc4 with OFFLINE state
2011-10-11 05:05:29,187 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for
e21b9e1545a28953aba0098fda5c9cd9 with OFFLINE state
2011-10-11 05:05:29,195 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for
5cd9f55eecd43d088bbd505f6795131f with OFFLINE state
2011-10-11 05:05:29,229 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for
db5f641452a70b09b85a92970e4198c7 with OFFLINE state
2011-10-11 05:05:29,237 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for
a7b20a653919e7f41bfb2ed349af7d21 with OFFLINE state
2011-10-11 05:05:29,253 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for
c9385619425f737eab1a6624d2e097a8 with OFFLINE state
// we cleaned the all the zk nodes.
2011-10-11 05:05:29,262 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Clean cluster startup.
Assigning userregions
2011-10-11 05:05:29,262 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:58198-0x132f23a9a380000 Deleting any existing unassigned nodes
2011-10-11 05:05:29,367 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 9 region(s)
across 1 server(s), retainAssignment=true
2011-10-11 05:05:29,369 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Timeout-on-RIT=9000
2011-10-11 05:05:29,369 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 9 region(s)
to C3S3,54366,1318323920153
2011-10-11 05:05:29,369 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done
2011-10-11 05:05:29,371 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:58198-0x132f23a9a380000 Async create of unassigned node for
771d63e9327383159553619a4f2dc74f with OFFLINE state
2011-10-11 05:05:29,371 INFO org.apache.hadoop.hbase.master.HMaster: Master
has completed initialization
2011-10-11 05:05:29,371 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:58198-0x132f23a9a380000 Async create of unassigned node for
3cf860dd323fe6360f571aeafc129f95 with OFFLINE state
2011-10-11 05:05:29,371 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:58198-0x132f23a9a380000 Async create of unassigned node for
4065350214452a9d5c55243c734bef08 with OFFLINE state
2011-10-11 05:05:29,371 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:58198-0x132f23a9a380000 Async create of unassigned node for
4e81613f82a39fc6e5e89f96e7b3ccc4 with OFFLINE state
2011-10-11 05:05:29,371 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:58198-0x132f23a9a380000 Async create of unassigned node for
e21b9e1545a28953aba0098fda5c9cd9 with OFFLINE state
2011-10-11 05:05:29,372 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:58198-0x132f23a9a380000 Async create of unassigned node for
5cd9f55eecd43d088bbd505f6795131f with OFFLINE state
2011-10-11 05:05:29,372 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:58198-0x132f23a9a380000 Async create of unassigned node for
db5f641452a70b09b85a92970e4198c7 with OFFLINE state
2011-10-11 05:05:29,372 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:58198-0x132f23a9a380000 Async create of unassigned node for
a7b20a653919e7f41bfb2ed349af7d21 with OFFLINE state
2011-10-11 05:05:29,372 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:58198-0x132f23a9a380000 Async create of unassigned node for
c9385619425f737eab1a6624d2e097a8 with OFFLINE state