You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Jameson Li <ho...@gmail.com> on 2011/03/21 12:30:37 UTC
avoid assigning hbase-access-operatation map/reduce slot to the tasktracker?
Hello,
I have a hadoop cluster that has 10 nodes. I use both of the
hdfs/mapred(0.20.2) and the hbase(0.20.6).
Every node is both of a datanode and a tasktracker.
But I just configured 9 nodes as the region server and the
zookeeper(HBASE_MANAGES_ZK=true).
And then when I run the map/reduce job that access the hbase, it will return
the below error.
Should I avoid assigning hbase-operatation map/reduce slot to the
tasktracker?
Or I have to change all of the nodes as the zookeeper node in all of the
datanode(or tasktracker?)?
*2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.3.1-942149, built on 05/07/2010 17:14 GMT*
*2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:host.name=aaa*
*2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.version=1.6.0_20*
*2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.vendor=Sun Microsystems Inc.*
*2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.home=....*
*2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.class.path=...*
*2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.library.path=...*
*2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=...*
*2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>*
*2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.name=Linux*
*2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.arch=amd64*
*2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.version=2.6.18-128.el5*
*2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.name=hadoop*
*2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.home=...*
*2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.dir=..*
*2011-03-21 17:17:39,885 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection, connectString=localhost:2181 sessionTimeout=60000
watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@6d372656
*
*2011-03-21 17:17:39,901 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server localhost/127.0.0.1:2181*
*......*
*2011-03-21 17:18:44,364 WARN org.apache.zookeeper.ClientCnxn: Session 0x0
for server null, unexpected error, closing socket connection and attempting
reconnect*
*java.net.ConnectException: Connection refused*
* at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)*
* at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)*
* at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)*
*2011-03-21 17:18:44,465 WARN
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create /hbase
-- check quorum servers, currently=localhost:2181*
*org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase*
* at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)*
* at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)*
* at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)*
* at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)*
* at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405)
*
* at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432)
*
* at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545)
*
* at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:964)
*
* at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:625)
*
* at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601)
*
* at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:670)
*
* at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:630)
*
* at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601)
*
* at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:670)
*
* at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:634)
*
* at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601)
*
* at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:134)*
* at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:112)*
* at com.mediav.hadoop.MVHourlyClickLogImportIntoHBaseMapper.setup(Unknown
Source)*
* at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)*
* at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)*
* at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)*
* at org.apache.hadoop.mapred.Child.main(Child.java:170)*
*2011-03-21 17:18:45,690 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server localhost/127.0.0.1:2181*
*2011-03-21 17:18:45,691 WARN org.apache.zookeeper.ClientCnxn: Session 0x0
for server null, unexpected error, closing socket connection and attempting
reconnect*
*java.net.ConnectException: Connection refused*
* at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)*
* at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)*
* at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)*
*2011-03-21 17:18:46,949 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server localhost/127.0.0.1:2181*
*2011-03-21 17:18:46,950 WARN org.apache.zookeeper.ClientCnxn: Session 0x0
for server null, unexpected error, closing socket connection and attempting
reconnect*
*java.net.ConnectException: Connection refused*
* at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)*
* at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)*
* at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)*
Thanks,
Jameson Li.
Re: avoid assigning hbase-access-operatation map/reduce slot to the tasktracker?
Posted by Jameson Li <ho...@gmail.com>.
Hello,
To Harsh J <qw...@gmail.com>:
I am sorry that I had said a wrong description.
I have not configured the hbase-env.sh to "HBASE_MANAGES_ZK=true", and its
really configuration is "#HBASE_MANAGES_ZK=true".
I just think its default value is "HBASE_MANAGES_ZK=true".
My hadoop version is basic on hadoop-0.20.2 release and have added the
patchs:HADOOP-4675,5745,MAPREDUCE-1070,551,1089.
My hbase version is the hbase-0.20.6.
And the below is all of the configuration.
Nine of the nodes have the hbase configuration:
hbase-env.sh:
export HBASE_HOME=/opt/hbase
export JAVA_HOME=/opt/jdk
export HBASE_OPTS="$HBASE_OPTS -XX:+HeapDumpOnOutOfMemoryError
-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode"
export HBASE_PID_DIR=/hadoop/pids
# export HBASE_MANAGES_ZK=true
hbase-site.xml:
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://node01:54310</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://node01:54310/hbase</value>
</property>
<property>
<name>hbase.master</name>
<value>node01:60010</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>node01,node02,node03,node04,node05,node06,node07,node08,node09</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
...
</configuration>
regionservers:
node01
node02
node03
node04
node05
node06
node07
node08
node09
And all of the nodes(10nodes) have the below hadoop(hdfs and mapred)
configration:
hadoop-env.sh:
export JAVA_HOME=/opt/jdk
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_JOBTRACKER_OPTS"
export HADOOP_PID_DIR=/hadoop/pids
mapred-site.xml:
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>node07:9001</value>
</property>
...
</configuration>
core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://node01:54310</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoop</value>
</property>
</configuration>
slaves:
node01
node02
node03
node04
node05
node06
node07
node08
node09
node10
masters:
node08
Every node is both of a datanode and a tasktracker.
But I just configured 9 nodes as the region server and the zookeeper.
And then when I run the map/reduce job that access the hbase, the node that
has not configured the regionserver and the zookeeper will return the below
error.
Should I avoid assigning hbase-operatation map/reduce slot to the
tasktracker?
Or I have to change all of the nodes as the zookeeper node in all of the
datanode(or tasktracker?)?
In the node10 had got the below error(And the others nodes are normal.):
2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.3.1-942149, built on 05/07/2010 17:14 GMT
2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:host.name=aaa
2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.version=1.6.0_20
2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.vendor=Sun Microsystems Inc.
2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.home=....
2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.class.path=...
2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.library.path=...
2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=...
2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.name=Linux
2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.version=2.6.18-128.el5
2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.name=hadoop
2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.home=...
2011-03-21 17:17:39,883 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.dir=..
2011-03-21 17:17:39,885 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection, connectString=localhost:2181 sessionTimeout=60000
watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@6d372656
2011-03-21 17:17:39,901 INFO org.apache.zookeeper.ClientCnxn: Opening socket
connection to server localhost/127.0.0.1:2181
......
2011-03-21 17:18:44,364 WARN org.apache.zookeeper.ClientCnxn: Session 0x0
for server null, unexpected error, closing socket connection and attempting
reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
2011-03-21 17:18:44,465 WARN
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create /hbase
-- check quorum servers, currently=localhost:2181
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:964)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:625)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:670)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:630)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:670)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:634)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:134)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:112)
at com.mediav.hadoop.MVHourlyClickLogImportIntoHBaseMapper.setup(Unknown
Source)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
2011-03-21 17:18:45,690 INFO org.apache.zookeeper.ClientCnxn: Opening socket
connection to server localhost/127.0.0.1:2181
2011-03-21 17:18:45,691 WARN org.apache.zookeeper.ClientCnxn: Session 0x0
for server null, unexpected error, closing socket connection and attempting
reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
2011-03-21 17:18:46,949 INFO org.apache.zookeeper.ClientCnxn: Opening socket
connection to server localhost/127.0.0.1:2181
2011-03-21 17:18:46,950 WARN org.apache.zookeeper.ClientCnxn: Session 0x0
for server null, unexpected error, closing socket connection and attempting
reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
Thanks,
Jameson Li.
Re: avoid assigning hbase-access-operatation map/reduce slot to the tasktracker?
Posted by Harsh J <qw...@gmail.com>.
On Mon, Mar 21, 2011 at 5:00 PM, Jameson Li <ho...@gmail.com> wrote:
> zookeeper(HBASE_MANAGES_ZK=true).
This and the line below point out that your 'configuration' of HBase
is somehow only half-ready for a fully-distributed operation :)
> connection to server localhost/127.0.0.1:2181
The issue is that, by default for easy setups of
local/pseudo-distributed modes, etc., HBase launches its managed
ZooKeeper instance to listen on 'localhost'. Your services and tasks
running elsewhere would never be able to connect to an 'lo' listening
service, of course.
You should change this behavior by setting the hbase.zookeeper.quorum
property to the proper, actual set of hosts you want ZooKeeper to run
on (In your case you could give it the HBase master's hostname,
perhaps? Or three of them.). The docs already speak of this: (see
hbase.zookeeper.quorum)
http://hbase.apache.org/docs/r0.20.6/hbase-conf.html :)
Note: It is always better to setup ZooKeeper by self, and externally
managed. Doesn't take more than 5m to configure and setup a
defaults-mostly cluster and not have HBase monitor it. :)
--
Harsh J
http://harshj.com