You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Vamshi Krishna <va...@gmail.com> on 2011/11/26 07:37:53 UTC
hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No such
file or directory
Hi i am running hbase on 3 machines, on one node master and regionserver,
on other two nodes regionservers. i ran bin/start-hbase.sh on master, all
the hbase daemons running on master node(HMaster, HQuorumpeer,
HRegionServer), But on the other regionserver machines i could not find any
hbase daemons running. According to my knowledge the HRegionserver daemons
should run on other machines automatically when i run bin/start-hbase.sh on
master node, right? But its not happening so.
More over its printing errors on my master node when i run
bin/start-hbase.sh
hbase-regionserver1: bash: line 0: cd: {HBASE_HOME}/bin/..: No such file or
directory
hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No such file
or directory
hbase-regionserver2: bash: line 0: cd: {HBASE_HOME}/bin/..: No such file or
directory
hbase-regionserver2: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No such file
or directory ,
but i could find out {HBASE_HOME}/bin/hbase-daemon.sh on both machines.
Infact the path of the {HBASE_HOME} folder on each of the respective
machines machines is same. i.e
/home/hduser/Documents/HBASE_SOFTWRAE/hbase-0.90.4
please can anybody help?
--
*Regards*
*
Vamshi Krishna
*
Re: hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No
such file or directory
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Sure no problem.
I understand that for some people this type of issue can be
frustrating. HBase being distributed by nature there's a lot more
going on than just setting up one mysql instance and it requires
skills/knowledge more attuned to devops rather than just devs.
Hopefully HBase will get better at identifying those issues for the
user. Right now it's just being stubborn and looks the other way :)
J-D
On Fri, Dec 2, 2011 at 11:01 PM, Vamshi Krishna <va...@gmail.com> wrote:
> Ohh Thank you very much J-D, its working fine. I am very glad for getting
> your continuous help for hbase successfull setup. Soon i want to start a
> blog in which i would post clear setup of hbase and problems along with
> corresponding solutions i faced during setting up all these days.
>
>
>
>
> On Fri, Dec 2, 2011 at 11:38 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> > i could not find 10.0.1.54:60000 any where on the console, rather i
>> found> 127.0.1.1:60000 in some places.
>>
>> Remove the 127.0.1.1 line in /etc/hosts, do it for all your ubuntu
>> machines.
>>
>> J-D
>> On Fri, Dec 2, 2011 at 6:34 AM, Vamshi Krishna <va...@gmail.com>
>> wrote:
>> > I disabled the firewall in all the machines.Then i started
>> > hbase(bin/start-hbase.sh), after 2-3 minutesi stopped hbase on master
>> node
>> > (bin/stop-hbase.sh). Even then HRegionserver daemon is running on region
>> > server machine. So i checked regionserver machines log, the full log is
>> > here
>> >
>> > http://pastebin.com/FXsNLMD9
>> >
>> > I found that regionserver machine is not able to communicate with master
>> > though it is attempting to connect to master (hbase-master:60000)for
>> every
>> > one minute or so.
>> >
>> > so i tried to do the following as you have told.
>> >
>> > hduser@vamshi-laptop:~$ telnet hbase-master 60000
>> > Trying 10.0.1.54...
>> > telnet: Unable to connect to remote host: Connection refused
>> >
>> > hduser@vamshi-laptop:~$ telnet hbase-master:60000
>> > telnet: could not resolve hbase-master:60000/telnet: Name or service not
>> > known
>> >
>> > i made hbase-master to bind to 10.0.1.54 in /etc/hosts file, so i
>> tried
>> > even following also.
>> >
>> > hduser@vamshi-laptop:~$ telnet 10.0.1.54 60000
>> > Trying 10.0.1.54...
>> > telnet: Unable to connect to remote host: Connection refused
>> > hduser@vamshi-laptop:~$
>> >
>> > So, i thought whether hbase-master is bound to port 60000 on the master
>> > node or not, so while hbase master is running, i checked by typing
>> > netstat -anp | grep 60000
>> >
>> > i could not find 10.0.1.54:60000 any where on the console, rather i
>> found
>> > 127.0.1.1:60000 in some places.
>> >
>> > Is this the correct way of representing /etc/hosts file on master node?
>> >
>> > 127.0.0.1 localhost
>> > 127.0.1.1 vamshikrishna-laptop
>> > 10.0.1.54 hbase-master hadoop-namenode vamshikrishna-desktop
>> > 10.0.1.55 hbase-regionserver1 hadoop-datanode1
>> > 10.0.1.97 hbase-regionserver2 hadoop-datanode2 arun-desktop
>> > # The following lines are desirable for IPv6 capable hosts
>> > ::1 localhost ip6-localhost ip6-loopback
>> > fe00::0 ip6-localnet
>> > ff00::0 ip6-mcastprefix
>> > ff02::1 ip6-allnodes
>> > ff02::2 ip6-allrouters
>> > ff02::3 ip6-allhosts
>> >
>> > Even i checked by typing the command on both master node and region
>> server
>> > machine
>> > $ nmap -p 60000 hbase-master
>> > then i got some thing on console like
>> > 60000/tcp closed
>> >
>> >
>> > Please help me, i dont understand what went wrong in my set up.
>> >
>> >
>> > On Thu, Dec 1, 2011 at 11:28 PM, Jean-Daniel Cryans <jdcryans@apache.org
>> >wrote:
>> >
>> >> So since I don't see the rest of the log I'll have to assume that the
>> >> region server was never able to connect to the master. Connection
>> >> refused could be a firewall, start the master and then try to telnet
>> >> from the other machines to master:60000.
>> >>
>> >> J-D
>> >>
>> >> On Thu, Dec 1, 2011 at 6:45 AM, Vamshi Krishna <va...@gmail.com>
>> >> wrote:
>> >> > I found in the logs of region server machines, i found this error (on
>> >> both
>> >> > regionserver machines)
>> >> >
>> >> > 2011-11-30 14:43:42,447 INFO org.apache.hadoop.ipc.HbaseRPC: Server at
>> >> > hbase-master/10.0.1.54:60020 could not be reached after 1 tries,
>> giving
>> >> up.
>> >> > *2011-11-30 14:44:37,762* WARN
>> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect
>> to
>> >> > master. Retrying. Error was:
>> >> > java.net.ConnectException: Connection refused
>> >> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> >> > at
>> >> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
>> >> > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
>> >> > at
>> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
>> >> > at
>> >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>> >> > at $Proxy5.getProtocolVersion(Unknown Source)
>> >> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
>> >> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
>> >> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
>> >> > at
>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1462)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
>> >> > at java.lang.Thread.run(Thread.java:662)
>> >> > 2011-11-30 14:44:40,768 INFO
>> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting
>> connect to
>> >> > Master server at hbase-master:60000
>> >> > *2011-11-30 14:45:40,847* WARN
>> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect
>> to
>> >> > master. Retrying. Error was:
>> >> > java.net.ConnectException: Connection refused
>> >> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> >> > at
>> >> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
>> >> > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
>> >> > at
>> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
>> >> > at
>> >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>> >> > at $Proxy5.getProtocolVersion(Unknown Source)
>> >> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
>> >> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
>> >> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
>> >> > at
>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1462)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
>> >> > at java.lang.Thread.run(Thread.java:662)
>> >> >
>> >> >
>> >> > and the same error is observed in the whole log repeatedly. After
>> seeing
>> >> it
>> >> > what i understand is that some how master started HRegionServer
>> daemons
>> >> on
>> >> > the machines but from then onwards the RegionServer machines are not
>> able
>> >> > to communicate with master. If we observe it is trying to communicate
>> >> with
>> >> > master for evry one minute.
>> >> >
>> >> > But i am not understanding where to check and modify the things..
>> please
>> >> > help. i feel all connections are OK.
>> >> >
>> >> > On Thu, Dec 1, 2011 at 12:28 AM, Jean-Daniel Cryans <
>> jdcryans@apache.org
>> >> >wrote:
>> >> >
>> >> >> stop-hbase.sh only tells the master to stop, which in turn will tell
>> >> >> the region servers to stop. If they are still running, it might be
>> >> >> because of an error. Look at their logs to figure what's going on.
>> >> >>
>> >> >> J-D
>> >> >>
>> >> >> On Tue, Nov 29, 2011 at 10:46 PM, Vamshi Krishna <
>> vamshi2105@gmail.com>
>> >> >> wrote:
>> >> >> > hey soryy for posting multiple times.
>> >> >> > J-D, As you said, i refered to my regionserver log, there i found
>> >> >> > Could not resolve the DNS name of
>> vamshikrishna-desktop
>> >> >> > so i added an alias ' vamshikrishna-desktop ' to its corresponding
>> IP
>> >> >> > address in /etc/hosts. So, from then master is able to run
>> >> HRegionServer
>> >> >> > daemon in the regionserver machines also.
>> >> >> >
>> >> >> > But the ONLY problem now is when i stop hbase on my master node by
>> >> >> running
>> >> >> > bin/stop-hbase.sh, all hbase daemons are stopping on matser node
>> but
>> >> NOT
>> >> >> on
>> >> >> > regionserver nodes.The HRegionServer daemon is still running on the
>> >> other
>> >> >> > regionserver machines.
>> >> >> > I think the HRegionServer daemons on all the regionserver machines
>> >> should
>> >> >> > also get stopped, when we execute bin/sto-hbase.sh on master node,
>> >> right?
>> >> >> > please help..
>> >> >> >
>> >> >> > On Wed, Nov 30, 2011 at 12:38 AM, Jean-Daniel Cryans <
>> >> >> jdcryans@apache.org>wrote:
>> >> >> >
>> >> >> >> You posted this in two threads, please refrain from doing this in
>> the
>> >> >> >> future.
>> >> >> >>
>> >> >> >> From what I can read in there, it tried to get the master address
>> >> >> >> that's supposed to be in zookeeper but it failed because it was
>> >> >> >> missing and then died. The way it's handled is a bit ugly but the
>> >> >> >> effect is the same, take a look at the master log to see what's
>> going
>> >> >> >> on.
>> >> >> >>
>> >> >> >> J-D
>> >> >> >>
>> >> >> >> On Mon, Nov 28, 2011 at 10:33 PM, Vamshi Krishna <
>> >> vamshi2105@gmail.com>
>> >> >> >> wrote:
>> >> >> >> > Hi Lars,
>> >> >> >> > i am not using cygwin, i am using 3 ubuntu-10.04 machines.
>> >> >> >> > Finally that problem i mentioned got resolved i.e now i can see
>> the
>> >> >> >> > following after i run bin/start-hbase.sh on my master machine,
>> >> >> >> >
>> >> >> >> > hbase-master: starting zookeeper, logging to
>> >> >> >> >
>> >> >> >>
>> >> >>
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-zookeeper-vamshikrishna-desktop.out
>> >> >> >> > starting master, logging to
>> >> >> >> >
>> >> >> >>
>> >> >>
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/logs/hbase-hduser-master-vamshikrishna-desktop.out
>> >> >> >> > hbase-regionserver2: starting regionserver, logging to
>> >> >> >> >
>> >> >> >>
>> >> >>
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.out
>> >> >> >> > hbase-regionserver1: starting regionserver, logging to
>> >> >> >> >
>> >> >> >>
>> >> >>
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.out
>> >> >> >> >
>> >> >> >> > hbase-master: starting regionserver, logging to
>> >> >> >> >
>> >> >> >>
>> >> >>
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshikrishna-desktop.out
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > But still , if i do jps on both region server machines,
>> >> HRegionServer
>> >> >> >> > daemon is not seen.And i checked the logs on both the
>> regionservere
>> >> >> >> > machines, i.e
>> >> >> >> >
>> >> >> >> >
>> >> >> >>
>> >> >>
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.log
>> >> >> >> > ---(hbase-regionserver1)
>> >> >> >> > and
>> >> >> >> >
>> >> >> >>
>> >> >>
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.log
>> >> >> >> > ---(hbase-regionserver2)
>> >> >> >> >
>> >> >> >> > i found the same in both .it is like this, (here
>> >> >> vamshikrishna-desktop
>> >> >> >> is
>> >> >> >> > the master and namenode machine)
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > 2011-11-29 11:24:36,714 ERROR
>> >> org.apache.hadoop.hbase.HServerAddress:
>> >> >> >> Could
>> >> >> >> > not resolve the DNS name of vamshikrishna-desktop
>> >> >> >> > 2011-11-29 11:24:36,716 FATAL
>> >> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING
>> region
>> >> >> >> server
>> >> >> >> > serverName=vamshi-laptop,60020,1322546076276, load=(requests=0,
>> >> >> >> regions=0,
>> >> >> >> > usedHeap=0, maxHeap=0): Unhandled exception: hostname can't be
>> null
>> >> >> >> > java.lang.IllegalArgumentException: hostname can't be null
>> >> >> >> > at
>> java.net.InetSocketAddress.<init>(InetSocketAddress.java:121)
>> >> >> >> > at
>> >> >> >> >
>> >> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.HServerAddress.getResolvedAddress(HServerAddress.java:108)
>> >> >> >> > at
>> >> >> >>
>> org.apache.hadoop.hbase.HServerAddress.<init>(HServerAddress.java:64)
>> >> >> >> > at
>> >> >> >> >
>> >> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:63)
>> >> >> >> > at
>> >> >> >> >
>> >> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getMasterAddress(HRegionServer.java:1484)
>> >> >> >> > at
>> >> >> >> >
>> >> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1457)
>> >> >> >> > at
>> >> >> >> >
>> >> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
>> >> >> >> > at
>> >> >> >> >
>> >> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
>> >> >> >> > at
>> >> >> >> >
>> >> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
>> >> >> >> > at java.lang.Thread.run(Thread.java:662)
>> >> >> >> > 2011-11-29 11:24:36,719 INFO
>> >> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED:
>> >> Unhandled
>> >> >> >> > exception: hostname can't be null
>> >> >> >> > 2011-11-29 11:24:36,719 INFO org.apache.hadoop.ipc.HBaseServer:
>> >> >> Stopping
>> >> >> >> > server on 60020
>> >> >> >> > 2011-11-29 11:24:36,749 INFO
>> >> >> >> > org.apache.hadoop.hbase.regionserver.StoreFile: Allocating
>> >> >> LruBlockCache
>> >> >> >> > with maximum size 199.4m
>> >> >> >> > 2011-11-29 11:24:36,775 INFO
>> >> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: aborting
>> server
>> >> >> at:
>> >> >> >> > vamshi-laptop,60020,1322546076276
>> >> >> >> > 2011-11-29 11:24:36,775 DEBUG
>> >> >> >> > org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog
>> >> >> tracker
>> >> >> >> > org.apache.hadoop.hbase.catalog.CatalogTracker@da2cef
>> >> >> >> > 2011-11-29 11:24:36,776 INFO
>> >> >> org.apache.hadoop.hbase.regionserver.Leases:
>> >> >> >> > regionserver60020 closing leases
>> >> >> >> > 2011-11-29 11:24:36,776 INFO
>> >> >> org.apache.hadoop.hbase.regionserver.Leases:
>> >> >> >> > regionserver60020 closed leases
>> >> >> >> > 2011-11-29 11:24:36,776 INFO
>> >> >> >> >
>> >> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
>> >> >> >> > Closed zookeeper sessionid=0x133ede31f3c0004
>> >> >> >> > 2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ClientCnxn:
>> >> >> EventThread
>> >> >> >> > shut down
>> >> >> >> > 2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ZooKeeper:
>> >> Session:
>> >> >> >> > 0x133ede31f3c0004 closed
>> >> >> >> > 2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ClientCnxn:
>> >> >> EventThread
>> >> >> >> > shut down
>> >> >> >> > 2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ZooKeeper:
>> >> Session:
>> >> >> >> > 0x133ede31f3c0005 closed
>> >> >> >> > 2011-11-29 11:24:36,792 INFO
>> >> >> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>> >> >> >> starting;
>> >> >> >> > hbase.shutdown.hook=true;
>> fsShutdownHook=Thread[Thread-14,5,main]
>> >> >> >> > 2011-11-29 11:24:36,792 INFO
>> >> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED:
>> >> Shutdown
>> >> >> >> hook
>> >> >> >> > 2011-11-29 11:24:36,792 INFO
>> >> >> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs
>> >> >> shutdown
>> >> >> >> > hook thread.
>> >> >> >> > 2011-11-29 11:24:36,792 INFO
>> >> >> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>> >> >> >> finished.
>> >> >> >> >
>> >> >> >> > what is the cause for problem here, please help.
>> >> >> >> >
>> >> >> >> > On Mon, Nov 28, 2011 at 7:28 PM, Lars George <
>> >> lars.george@gmail.com>
>> >> >> >> wrote:
>> >> >> >> >
>> >> >> >> >> Hi,
>> >> >> >> >>
>> >> >> >> >> Did you add the list of servers to the regionservers file in
>> the
>> >> >> >> >> $HBASE_HOME/conf/ dir? Are you using Cygwin? Or what else is
>> your
>> >> >> >> >> environment?
>> >> >> >> >>
>> >> >> >> >> Lars
>> >> >> >> >>
>> >> >> >> >> On Nov 26, 2011, at 7:37 AM, Vamshi Krishna wrote:
>> >> >> >> >>
>> >> >> >> >> > Hi i am running hbase on 3 machines, on one node master and
>> >> >> >> regionserver,
>> >> >> >> >> > on other two nodes regionservers. i ran bin/start-hbase.sh on
>> >> >> master,
>> >> >> >> all
>> >> >> >> >> > the hbase daemons running on master node(HMaster,
>> HQuorumpeer,
>> >> >> >> >> > HRegionServer), But on the other regionserver machines i
>> could
>> >> not
>> >> >> >> find
>> >> >> >> >> any
>> >> >> >> >> > hbase daemons running. According to my knowledge the
>> >> HRegionserver
>> >> >> >> >> daemons
>> >> >> >> >> > should run on other machines automatically when i run
>> >> >> >> bin/start-hbase.sh
>> >> >> >> >> on
>> >> >> >> >> > master node, right? But its not happening so.
>> >> >> >> >> > More over its printing errors on my master node when i run
>> >> >> >> >> > bin/start-hbase.sh
>> >> >> >> >> >
>> >> >> >> >> > hbase-regionserver1: bash: line 0: cd: {HBASE_HOME}/bin/..:
>> No
>> >> such
>> >> >> >> file
>> >> >> >> >> or
>> >> >> >> >> > directory
>> >> >> >> >> > hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh:
>> No
>> >> >> such
>> >> >> >> file
>> >> >> >> >> > or directory
>> >> >> >> >> > hbase-regionserver2: bash: line 0: cd: {HBASE_HOME}/bin/..:
>> No
>> >> such
>> >> >> >> file
>> >> >> >> >> or
>> >> >> >> >> > directory
>> >> >> >> >> > hbase-regionserver2: bash: {HBASE_HOME}/bin/hbase-daemon.sh:
>> No
>> >> >> such
>> >> >> >> file
>> >> >> >> >> > or directory ,
>> >> >> >> >> >
>> >> >> >> >> > but i could find out {HBASE_HOME}/bin/hbase-daemon.sh on both
>> >> >> >> machines.
>> >> >> >> >> > Infact the path of the {HBASE_HOME} folder on each of the
>> >> >> respective
>> >> >> >> >> > machines machines is same. i.e
>> >> >> >> >> > /home/hduser/Documents/HBASE_SOFTWRAE/hbase-0.90.4
>> >> >> >> >> >
>> >> >> >> >> > please can anybody help?
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > --
>> >> >> >> >> > *Regards*
>> >> >> >> >> > *
>> >> >> >> >> > Vamshi Krishna
>> >> >> >> >> > *
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > --
>> >> >> >> > *Regards*
>> >> >> >> > *
>> >> >> >> > Vamshi Krishna
>> >> >> >> > *
>> >> >> >> >
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > *Regards*
>> >> >> > *
>> >> >> > Vamshi Krishna
>> >> >> > *
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > *Regards*
>> >> > *
>> >> > Vamshi Krishna
>> >> > *
>> >>
>> >
>> >
>> >
>> > --
>> > *Regards*
>> > *
>> > Vamshi Krishna
>> > *
>>
>
>
>
> --
> *Regards*
> *
> Vamshi Krishna
> *
Re: hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No
such file or directory
Posted by Vamshi Krishna <va...@gmail.com>.
Ohh Thank you very much J-D, its working fine. I am very glad for getting
your continuous help for hbase successfull setup. Soon i want to start a
blog in which i would post clear setup of hbase and problems along with
corresponding solutions i faced during setting up all these days.
On Fri, Dec 2, 2011 at 11:38 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> > i could not find 10.0.1.54:60000 any where on the console, rather i
> found> 127.0.1.1:60000 in some places.
>
> Remove the 127.0.1.1 line in /etc/hosts, do it for all your ubuntu
> machines.
>
> J-D
> On Fri, Dec 2, 2011 at 6:34 AM, Vamshi Krishna <va...@gmail.com>
> wrote:
> > I disabled the firewall in all the machines.Then i started
> > hbase(bin/start-hbase.sh), after 2-3 minutesi stopped hbase on master
> node
> > (bin/stop-hbase.sh). Even then HRegionserver daemon is running on region
> > server machine. So i checked regionserver machines log, the full log is
> > here
> >
> > http://pastebin.com/FXsNLMD9
> >
> > I found that regionserver machine is not able to communicate with master
> > though it is attempting to connect to master (hbase-master:60000)for
> every
> > one minute or so.
> >
> > so i tried to do the following as you have told.
> >
> > hduser@vamshi-laptop:~$ telnet hbase-master 60000
> > Trying 10.0.1.54...
> > telnet: Unable to connect to remote host: Connection refused
> >
> > hduser@vamshi-laptop:~$ telnet hbase-master:60000
> > telnet: could not resolve hbase-master:60000/telnet: Name or service not
> > known
> >
> > i made hbase-master to bind to 10.0.1.54 in /etc/hosts file, so i
> tried
> > even following also.
> >
> > hduser@vamshi-laptop:~$ telnet 10.0.1.54 60000
> > Trying 10.0.1.54...
> > telnet: Unable to connect to remote host: Connection refused
> > hduser@vamshi-laptop:~$
> >
> > So, i thought whether hbase-master is bound to port 60000 on the master
> > node or not, so while hbase master is running, i checked by typing
> > netstat -anp | grep 60000
> >
> > i could not find 10.0.1.54:60000 any where on the console, rather i
> found
> > 127.0.1.1:60000 in some places.
> >
> > Is this the correct way of representing /etc/hosts file on master node?
> >
> > 127.0.0.1 localhost
> > 127.0.1.1 vamshikrishna-laptop
> > 10.0.1.54 hbase-master hadoop-namenode vamshikrishna-desktop
> > 10.0.1.55 hbase-regionserver1 hadoop-datanode1
> > 10.0.1.97 hbase-regionserver2 hadoop-datanode2 arun-desktop
> > # The following lines are desirable for IPv6 capable hosts
> > ::1 localhost ip6-localhost ip6-loopback
> > fe00::0 ip6-localnet
> > ff00::0 ip6-mcastprefix
> > ff02::1 ip6-allnodes
> > ff02::2 ip6-allrouters
> > ff02::3 ip6-allhosts
> >
> > Even i checked by typing the command on both master node and region
> server
> > machine
> > $ nmap -p 60000 hbase-master
> > then i got some thing on console like
> > 60000/tcp closed
> >
> >
> > Please help me, i dont understand what went wrong in my set up.
> >
> >
> > On Thu, Dec 1, 2011 at 11:28 PM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
> >
> >> So since I don't see the rest of the log I'll have to assume that the
> >> region server was never able to connect to the master. Connection
> >> refused could be a firewall, start the master and then try to telnet
> >> from the other machines to master:60000.
> >>
> >> J-D
> >>
> >> On Thu, Dec 1, 2011 at 6:45 AM, Vamshi Krishna <va...@gmail.com>
> >> wrote:
> >> > I found in the logs of region server machines, i found this error (on
> >> both
> >> > regionserver machines)
> >> >
> >> > 2011-11-30 14:43:42,447 INFO org.apache.hadoop.ipc.HbaseRPC: Server at
> >> > hbase-master/10.0.1.54:60020 could not be reached after 1 tries,
> giving
> >> up.
> >> > *2011-11-30 14:44:37,762* WARN
> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect
> to
> >> > master. Retrying. Error was:
> >> > java.net.ConnectException: Connection refused
> >> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >> > at
> >> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> >> > at
> >> >
> >>
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> >> > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
> >> > at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
> >> > at
> >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> >> > at $Proxy5.getProtocolVersion(Unknown Source)
> >> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
> >> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
> >> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
> >> > at
> >> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1462)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
> >> > at java.lang.Thread.run(Thread.java:662)
> >> > 2011-11-30 14:44:40,768 INFO
> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting
> connect to
> >> > Master server at hbase-master:60000
> >> > *2011-11-30 14:45:40,847* WARN
> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect
> to
> >> > master. Retrying. Error was:
> >> > java.net.ConnectException: Connection refused
> >> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >> > at
> >> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> >> > at
> >> >
> >>
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> >> > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
> >> > at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
> >> > at
> >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> >> > at $Proxy5.getProtocolVersion(Unknown Source)
> >> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
> >> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
> >> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
> >> > at
> >> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1462)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
> >> > at java.lang.Thread.run(Thread.java:662)
> >> >
> >> >
> >> > and the same error is observed in the whole log repeatedly. After
> seeing
> >> it
> >> > what i understand is that some how master started HRegionServer
> daemons
> >> on
> >> > the machines but from then onwards the RegionServer machines are not
> able
> >> > to communicate with master. If we observe it is trying to communicate
> >> with
> >> > master for evry one minute.
> >> >
> >> > But i am not understanding where to check and modify the things..
> please
> >> > help. i feel all connections are OK.
> >> >
> >> > On Thu, Dec 1, 2011 at 12:28 AM, Jean-Daniel Cryans <
> jdcryans@apache.org
> >> >wrote:
> >> >
> >> >> stop-hbase.sh only tells the master to stop, which in turn will tell
> >> >> the region servers to stop. If they are still running, it might be
> >> >> because of an error. Look at their logs to figure what's going on.
> >> >>
> >> >> J-D
> >> >>
> >> >> On Tue, Nov 29, 2011 at 10:46 PM, Vamshi Krishna <
> vamshi2105@gmail.com>
> >> >> wrote:
> >> >> > hey soryy for posting multiple times.
> >> >> > J-D, As you said, i refered to my regionserver log, there i found
> >> >> > Could not resolve the DNS name of
> vamshikrishna-desktop
> >> >> > so i added an alias ' vamshikrishna-desktop ' to its corresponding
> IP
> >> >> > address in /etc/hosts. So, from then master is able to run
> >> HRegionServer
> >> >> > daemon in the regionserver machines also.
> >> >> >
> >> >> > But the ONLY problem now is when i stop hbase on my master node by
> >> >> running
> >> >> > bin/stop-hbase.sh, all hbase daemons are stopping on matser node
> but
> >> NOT
> >> >> on
> >> >> > regionserver nodes.The HRegionServer daemon is still running on the
> >> other
> >> >> > regionserver machines.
> >> >> > I think the HRegionServer daemons on all the regionserver machines
> >> should
> >> >> > also get stopped, when we execute bin/sto-hbase.sh on master node,
> >> right?
> >> >> > please help..
> >> >> >
> >> >> > On Wed, Nov 30, 2011 at 12:38 AM, Jean-Daniel Cryans <
> >> >> jdcryans@apache.org>wrote:
> >> >> >
> >> >> >> You posted this in two threads, please refrain from doing this in
> the
> >> >> >> future.
> >> >> >>
> >> >> >> From what I can read in there, it tried to get the master address
> >> >> >> that's supposed to be in zookeeper but it failed because it was
> >> >> >> missing and then died. The way it's handled is a bit ugly but the
> >> >> >> effect is the same, take a look at the master log to see what's
> going
> >> >> >> on.
> >> >> >>
> >> >> >> J-D
> >> >> >>
> >> >> >> On Mon, Nov 28, 2011 at 10:33 PM, Vamshi Krishna <
> >> vamshi2105@gmail.com>
> >> >> >> wrote:
> >> >> >> > Hi Lars,
> >> >> >> > i am not using cygwin, i am using 3 ubuntu-10.04 machines.
> >> >> >> > Finally that problem i mentioned got resolved i.e now i can see
> the
> >> >> >> > following after i run bin/start-hbase.sh on my master machine,
> >> >> >> >
> >> >> >> > hbase-master: starting zookeeper, logging to
> >> >> >> >
> >> >> >>
> >> >>
> >>
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-zookeeper-vamshikrishna-desktop.out
> >> >> >> > starting master, logging to
> >> >> >> >
> >> >> >>
> >> >>
> >>
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/logs/hbase-hduser-master-vamshikrishna-desktop.out
> >> >> >> > hbase-regionserver2: starting regionserver, logging to
> >> >> >> >
> >> >> >>
> >> >>
> >>
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.out
> >> >> >> > hbase-regionserver1: starting regionserver, logging to
> >> >> >> >
> >> >> >>
> >> >>
> >>
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.out
> >> >> >> >
> >> >> >> > hbase-master: starting regionserver, logging to
> >> >> >> >
> >> >> >>
> >> >>
> >>
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshikrishna-desktop.out
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > But still , if i do jps on both region server machines,
> >> HRegionServer
> >> >> >> > daemon is not seen.And i checked the logs on both the
> regionservere
> >> >> >> > machines, i.e
> >> >> >> >
> >> >> >> >
> >> >> >>
> >> >>
> >>
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.log
> >> >> >> > ---(hbase-regionserver1)
> >> >> >> > and
> >> >> >> >
> >> >> >>
> >> >>
> >>
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.log
> >> >> >> > ---(hbase-regionserver2)
> >> >> >> >
> >> >> >> > i found the same in both .it is like this, (here
> >> >> vamshikrishna-desktop
> >> >> >> is
> >> >> >> > the master and namenode machine)
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > 2011-11-29 11:24:36,714 ERROR
> >> org.apache.hadoop.hbase.HServerAddress:
> >> >> >> Could
> >> >> >> > not resolve the DNS name of vamshikrishna-desktop
> >> >> >> > 2011-11-29 11:24:36,716 FATAL
> >> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING
> region
> >> >> >> server
> >> >> >> > serverName=vamshi-laptop,60020,1322546076276, load=(requests=0,
> >> >> >> regions=0,
> >> >> >> > usedHeap=0, maxHeap=0): Unhandled exception: hostname can't be
> null
> >> >> >> > java.lang.IllegalArgumentException: hostname can't be null
> >> >> >> > at
> java.net.InetSocketAddress.<init>(InetSocketAddress.java:121)
> >> >> >> > at
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.HServerAddress.getResolvedAddress(HServerAddress.java:108)
> >> >> >> > at
> >> >> >>
> org.apache.hadoop.hbase.HServerAddress.<init>(HServerAddress.java:64)
> >> >> >> > at
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:63)
> >> >> >> > at
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMasterAddress(HRegionServer.java:1484)
> >> >> >> > at
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1457)
> >> >> >> > at
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
> >> >> >> > at
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
> >> >> >> > at
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
> >> >> >> > at java.lang.Thread.run(Thread.java:662)
> >> >> >> > 2011-11-29 11:24:36,719 INFO
> >> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED:
> >> Unhandled
> >> >> >> > exception: hostname can't be null
> >> >> >> > 2011-11-29 11:24:36,719 INFO org.apache.hadoop.ipc.HBaseServer:
> >> >> Stopping
> >> >> >> > server on 60020
> >> >> >> > 2011-11-29 11:24:36,749 INFO
> >> >> >> > org.apache.hadoop.hbase.regionserver.StoreFile: Allocating
> >> >> LruBlockCache
> >> >> >> > with maximum size 199.4m
> >> >> >> > 2011-11-29 11:24:36,775 INFO
> >> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: aborting
> server
> >> >> at:
> >> >> >> > vamshi-laptop,60020,1322546076276
> >> >> >> > 2011-11-29 11:24:36,775 DEBUG
> >> >> >> > org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog
> >> >> tracker
> >> >> >> > org.apache.hadoop.hbase.catalog.CatalogTracker@da2cef
> >> >> >> > 2011-11-29 11:24:36,776 INFO
> >> >> org.apache.hadoop.hbase.regionserver.Leases:
> >> >> >> > regionserver60020 closing leases
> >> >> >> > 2011-11-29 11:24:36,776 INFO
> >> >> org.apache.hadoop.hbase.regionserver.Leases:
> >> >> >> > regionserver60020 closed leases
> >> >> >> > 2011-11-29 11:24:36,776 INFO
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> >> >> >> > Closed zookeeper sessionid=0x133ede31f3c0004
> >> >> >> > 2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ClientCnxn:
> >> >> EventThread
> >> >> >> > shut down
> >> >> >> > 2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ZooKeeper:
> >> Session:
> >> >> >> > 0x133ede31f3c0004 closed
> >> >> >> > 2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ClientCnxn:
> >> >> EventThread
> >> >> >> > shut down
> >> >> >> > 2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ZooKeeper:
> >> Session:
> >> >> >> > 0x133ede31f3c0005 closed
> >> >> >> > 2011-11-29 11:24:36,792 INFO
> >> >> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
> >> >> >> starting;
> >> >> >> > hbase.shutdown.hook=true;
> fsShutdownHook=Thread[Thread-14,5,main]
> >> >> >> > 2011-11-29 11:24:36,792 INFO
> >> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED:
> >> Shutdown
> >> >> >> hook
> >> >> >> > 2011-11-29 11:24:36,792 INFO
> >> >> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs
> >> >> shutdown
> >> >> >> > hook thread.
> >> >> >> > 2011-11-29 11:24:36,792 INFO
> >> >> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
> >> >> >> finished.
> >> >> >> >
> >> >> >> > what is the cause for problem here, please help.
> >> >> >> >
> >> >> >> > On Mon, Nov 28, 2011 at 7:28 PM, Lars George <
> >> lars.george@gmail.com>
> >> >> >> wrote:
> >> >> >> >
> >> >> >> >> Hi,
> >> >> >> >>
> >> >> >> >> Did you add the list of servers to the regionservers file in
> the
> >> >> >> >> $HBASE_HOME/conf/ dir? Are you using Cygwin? Or what else is
> your
> >> >> >> >> environment?
> >> >> >> >>
> >> >> >> >> Lars
> >> >> >> >>
> >> >> >> >> On Nov 26, 2011, at 7:37 AM, Vamshi Krishna wrote:
> >> >> >> >>
> >> >> >> >> > Hi i am running hbase on 3 machines, on one node master and
> >> >> >> regionserver,
> >> >> >> >> > on other two nodes regionservers. i ran bin/start-hbase.sh on
> >> >> master,
> >> >> >> all
> >> >> >> >> > the hbase daemons running on master node(HMaster,
> HQuorumpeer,
> >> >> >> >> > HRegionServer), But on the other regionserver machines i
> could
> >> not
> >> >> >> find
> >> >> >> >> any
> >> >> >> >> > hbase daemons running. According to my knowledge the
> >> HRegionserver
> >> >> >> >> daemons
> >> >> >> >> > should run on other machines automatically when i run
> >> >> >> bin/start-hbase.sh
> >> >> >> >> on
> >> >> >> >> > master node, right? But its not happening so.
> >> >> >> >> > More over its printing errors on my master node when i run
> >> >> >> >> > bin/start-hbase.sh
> >> >> >> >> >
> >> >> >> >> > hbase-regionserver1: bash: line 0: cd: {HBASE_HOME}/bin/..:
> No
> >> such
> >> >> >> file
> >> >> >> >> or
> >> >> >> >> > directory
> >> >> >> >> > hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh:
> No
> >> >> such
> >> >> >> file
> >> >> >> >> > or directory
> >> >> >> >> > hbase-regionserver2: bash: line 0: cd: {HBASE_HOME}/bin/..:
> No
> >> such
> >> >> >> file
> >> >> >> >> or
> >> >> >> >> > directory
> >> >> >> >> > hbase-regionserver2: bash: {HBASE_HOME}/bin/hbase-daemon.sh:
> No
> >> >> such
> >> >> >> file
> >> >> >> >> > or directory ,
> >> >> >> >> >
> >> >> >> >> > but i could find out {HBASE_HOME}/bin/hbase-daemon.sh on both
> >> >> >> machines.
> >> >> >> >> > Infact the path of the {HBASE_HOME} folder on each of the
> >> >> respective
> >> >> >> >> > machines machines is same. i.e
> >> >> >> >> > /home/hduser/Documents/HBASE_SOFTWRAE/hbase-0.90.4
> >> >> >> >> >
> >> >> >> >> > please can anybody help?
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > --
> >> >> >> >> > *Regards*
> >> >> >> >> > *
> >> >> >> >> > Vamshi Krishna
> >> >> >> >> > *
> >> >> >> >>
> >> >> >> >>
> >> >> >> >
> >> >> >> >
> >> >> >> > --
> >> >> >> > *Regards*
> >> >> >> > *
> >> >> >> > Vamshi Krishna
> >> >> >> > *
> >> >> >> >
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > *Regards*
> >> >> > *
> >> >> > Vamshi Krishna
> >> >> > *
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > *Regards*
> >> > *
> >> > Vamshi Krishna
> >> > *
> >>
> >
> >
> >
> > --
> > *Regards*
> > *
> > Vamshi Krishna
> > *
>
--
*Regards*
*
Vamshi Krishna
*
Re: hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No
such file or directory
Posted by Jean-Daniel Cryans <jd...@apache.org>.
> i could not find 10.0.1.54:60000 any where on the console, rather i found> 127.0.1.1:60000 in some places.
Remove the 127.0.1.1 line in /etc/hosts, do it for all your ubuntu machines.
J-D
On Fri, Dec 2, 2011 at 6:34 AM, Vamshi Krishna <va...@gmail.com> wrote:
> I disabled the firewall in all the machines.Then i started
> hbase(bin/start-hbase.sh), after 2-3 minutesi stopped hbase on master node
> (bin/stop-hbase.sh). Even then HRegionserver daemon is running on region
> server machine. So i checked regionserver machines log, the full log is
> here
>
> http://pastebin.com/FXsNLMD9
>
> I found that regionserver machine is not able to communicate with master
> though it is attempting to connect to master (hbase-master:60000)for every
> one minute or so.
>
> so i tried to do the following as you have told.
>
> hduser@vamshi-laptop:~$ telnet hbase-master 60000
> Trying 10.0.1.54...
> telnet: Unable to connect to remote host: Connection refused
>
> hduser@vamshi-laptop:~$ telnet hbase-master:60000
> telnet: could not resolve hbase-master:60000/telnet: Name or service not
> known
>
> i made hbase-master to bind to 10.0.1.54 in /etc/hosts file, so i tried
> even following also.
>
> hduser@vamshi-laptop:~$ telnet 10.0.1.54 60000
> Trying 10.0.1.54...
> telnet: Unable to connect to remote host: Connection refused
> hduser@vamshi-laptop:~$
>
> So, i thought whether hbase-master is bound to port 60000 on the master
> node or not, so while hbase master is running, i checked by typing
> netstat -anp | grep 60000
>
> i could not find 10.0.1.54:60000 any where on the console, rather i found
> 127.0.1.1:60000 in some places.
>
> Is this the correct way of representing /etc/hosts file on master node?
>
> 127.0.0.1 localhost
> 127.0.1.1 vamshikrishna-laptop
> 10.0.1.54 hbase-master hadoop-namenode vamshikrishna-desktop
> 10.0.1.55 hbase-regionserver1 hadoop-datanode1
> 10.0.1.97 hbase-regionserver2 hadoop-datanode2 arun-desktop
> # The following lines are desirable for IPv6 capable hosts
> ::1 localhost ip6-localhost ip6-loopback
> fe00::0 ip6-localnet
> ff00::0 ip6-mcastprefix
> ff02::1 ip6-allnodes
> ff02::2 ip6-allrouters
> ff02::3 ip6-allhosts
>
> Even i checked by typing the command on both master node and region server
> machine
> $ nmap -p 60000 hbase-master
> then i got some thing on console like
> 60000/tcp closed
>
>
> Please help me, i dont understand what went wrong in my set up.
>
>
> On Thu, Dec 1, 2011 at 11:28 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> So since I don't see the rest of the log I'll have to assume that the
>> region server was never able to connect to the master. Connection
>> refused could be a firewall, start the master and then try to telnet
>> from the other machines to master:60000.
>>
>> J-D
>>
>> On Thu, Dec 1, 2011 at 6:45 AM, Vamshi Krishna <va...@gmail.com>
>> wrote:
>> > I found in the logs of region server machines, i found this error (on
>> both
>> > regionserver machines)
>> >
>> > 2011-11-30 14:43:42,447 INFO org.apache.hadoop.ipc.HbaseRPC: Server at
>> > hbase-master/10.0.1.54:60020 could not be reached after 1 tries, giving
>> up.
>> > *2011-11-30 14:44:37,762* WARN
>> > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to
>> > master. Retrying. Error was:
>> > java.net.ConnectException: Connection refused
>> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> > at
>> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
>> > at
>> >
>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
>> > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
>> > at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
>> > at
>> > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>> > at $Proxy5.getProtocolVersion(Unknown Source)
>> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
>> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
>> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1462)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
>> > at java.lang.Thread.run(Thread.java:662)
>> > 2011-11-30 14:44:40,768 INFO
>> > org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to
>> > Master server at hbase-master:60000
>> > *2011-11-30 14:45:40,847* WARN
>> > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to
>> > master. Retrying. Error was:
>> > java.net.ConnectException: Connection refused
>> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> > at
>> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
>> > at
>> >
>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
>> > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
>> > at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
>> > at
>> > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>> > at $Proxy5.getProtocolVersion(Unknown Source)
>> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
>> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
>> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1462)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
>> > at java.lang.Thread.run(Thread.java:662)
>> >
>> >
>> > and the same error is observed in the whole log repeatedly. After seeing
>> it
>> > what i understand is that some how master started HRegionServer daemons
>> on
>> > the machines but from then onwards the RegionServer machines are not able
>> > to communicate with master. If we observe it is trying to communicate
>> with
>> > master for evry one minute.
>> >
>> > But i am not understanding where to check and modify the things.. please
>> > help. i feel all connections are OK.
>> >
>> > On Thu, Dec 1, 2011 at 12:28 AM, Jean-Daniel Cryans <jdcryans@apache.org
>> >wrote:
>> >
>> >> stop-hbase.sh only tells the master to stop, which in turn will tell
>> >> the region servers to stop. If they are still running, it might be
>> >> because of an error. Look at their logs to figure what's going on.
>> >>
>> >> J-D
>> >>
>> >> On Tue, Nov 29, 2011 at 10:46 PM, Vamshi Krishna <va...@gmail.com>
>> >> wrote:
>> >> > hey soryy for posting multiple times.
>> >> > J-D, As you said, i refered to my regionserver log, there i found
>> >> > Could not resolve the DNS name of vamshikrishna-desktop
>> >> > so i added an alias ' vamshikrishna-desktop ' to its corresponding IP
>> >> > address in /etc/hosts. So, from then master is able to run
>> HRegionServer
>> >> > daemon in the regionserver machines also.
>> >> >
>> >> > But the ONLY problem now is when i stop hbase on my master node by
>> >> running
>> >> > bin/stop-hbase.sh, all hbase daemons are stopping on matser node but
>> NOT
>> >> on
>> >> > regionserver nodes.The HRegionServer daemon is still running on the
>> other
>> >> > regionserver machines.
>> >> > I think the HRegionServer daemons on all the regionserver machines
>> should
>> >> > also get stopped, when we execute bin/sto-hbase.sh on master node,
>> right?
>> >> > please help..
>> >> >
>> >> > On Wed, Nov 30, 2011 at 12:38 AM, Jean-Daniel Cryans <
>> >> jdcryans@apache.org>wrote:
>> >> >
>> >> >> You posted this in two threads, please refrain from doing this in the
>> >> >> future.
>> >> >>
>> >> >> From what I can read in there, it tried to get the master address
>> >> >> that's supposed to be in zookeeper but it failed because it was
>> >> >> missing and then died. The way it's handled is a bit ugly but the
>> >> >> effect is the same, take a look at the master log to see what's going
>> >> >> on.
>> >> >>
>> >> >> J-D
>> >> >>
>> >> >> On Mon, Nov 28, 2011 at 10:33 PM, Vamshi Krishna <
>> vamshi2105@gmail.com>
>> >> >> wrote:
>> >> >> > Hi Lars,
>> >> >> > i am not using cygwin, i am using 3 ubuntu-10.04 machines.
>> >> >> > Finally that problem i mentioned got resolved i.e now i can see the
>> >> >> > following after i run bin/start-hbase.sh on my master machine,
>> >> >> >
>> >> >> > hbase-master: starting zookeeper, logging to
>> >> >> >
>> >> >>
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-zookeeper-vamshikrishna-desktop.out
>> >> >> > starting master, logging to
>> >> >> >
>> >> >>
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/logs/hbase-hduser-master-vamshikrishna-desktop.out
>> >> >> > hbase-regionserver2: starting regionserver, logging to
>> >> >> >
>> >> >>
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.out
>> >> >> > hbase-regionserver1: starting regionserver, logging to
>> >> >> >
>> >> >>
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.out
>> >> >> >
>> >> >> > hbase-master: starting regionserver, logging to
>> >> >> >
>> >> >>
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshikrishna-desktop.out
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > But still , if i do jps on both region server machines,
>> HRegionServer
>> >> >> > daemon is not seen.And i checked the logs on both the regionservere
>> >> >> > machines, i.e
>> >> >> >
>> >> >> >
>> >> >>
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.log
>> >> >> > ---(hbase-regionserver1)
>> >> >> > and
>> >> >> >
>> >> >>
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.log
>> >> >> > ---(hbase-regionserver2)
>> >> >> >
>> >> >> > i found the same in both .it is like this, (here
>> >> vamshikrishna-desktop
>> >> >> is
>> >> >> > the master and namenode machine)
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > 2011-11-29 11:24:36,714 ERROR
>> org.apache.hadoop.hbase.HServerAddress:
>> >> >> Could
>> >> >> > not resolve the DNS name of vamshikrishna-desktop
>> >> >> > 2011-11-29 11:24:36,716 FATAL
>> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region
>> >> >> server
>> >> >> > serverName=vamshi-laptop,60020,1322546076276, load=(requests=0,
>> >> >> regions=0,
>> >> >> > usedHeap=0, maxHeap=0): Unhandled exception: hostname can't be null
>> >> >> > java.lang.IllegalArgumentException: hostname can't be null
>> >> >> > at java.net.InetSocketAddress.<init>(InetSocketAddress.java:121)
>> >> >> > at
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.hbase.HServerAddress.getResolvedAddress(HServerAddress.java:108)
>> >> >> > at
>> >> >> org.apache.hadoop.hbase.HServerAddress.<init>(HServerAddress.java:64)
>> >> >> > at
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.hbase.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:63)
>> >> >> > at
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getMasterAddress(HRegionServer.java:1484)
>> >> >> > at
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1457)
>> >> >> > at
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
>> >> >> > at
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
>> >> >> > at
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
>> >> >> > at java.lang.Thread.run(Thread.java:662)
>> >> >> > 2011-11-29 11:24:36,719 INFO
>> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED:
>> Unhandled
>> >> >> > exception: hostname can't be null
>> >> >> > 2011-11-29 11:24:36,719 INFO org.apache.hadoop.ipc.HBaseServer:
>> >> Stopping
>> >> >> > server on 60020
>> >> >> > 2011-11-29 11:24:36,749 INFO
>> >> >> > org.apache.hadoop.hbase.regionserver.StoreFile: Allocating
>> >> LruBlockCache
>> >> >> > with maximum size 199.4m
>> >> >> > 2011-11-29 11:24:36,775 INFO
>> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server
>> >> at:
>> >> >> > vamshi-laptop,60020,1322546076276
>> >> >> > 2011-11-29 11:24:36,775 DEBUG
>> >> >> > org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog
>> >> tracker
>> >> >> > org.apache.hadoop.hbase.catalog.CatalogTracker@da2cef
>> >> >> > 2011-11-29 11:24:36,776 INFO
>> >> org.apache.hadoop.hbase.regionserver.Leases:
>> >> >> > regionserver60020 closing leases
>> >> >> > 2011-11-29 11:24:36,776 INFO
>> >> org.apache.hadoop.hbase.regionserver.Leases:
>> >> >> > regionserver60020 closed leases
>> >> >> > 2011-11-29 11:24:36,776 INFO
>> >> >> >
>> >> >>
>> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
>> >> >> > Closed zookeeper sessionid=0x133ede31f3c0004
>> >> >> > 2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ClientCnxn:
>> >> EventThread
>> >> >> > shut down
>> >> >> > 2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ZooKeeper:
>> Session:
>> >> >> > 0x133ede31f3c0004 closed
>> >> >> > 2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ClientCnxn:
>> >> EventThread
>> >> >> > shut down
>> >> >> > 2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ZooKeeper:
>> Session:
>> >> >> > 0x133ede31f3c0005 closed
>> >> >> > 2011-11-29 11:24:36,792 INFO
>> >> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>> >> >> starting;
>> >> >> > hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-14,5,main]
>> >> >> > 2011-11-29 11:24:36,792 INFO
>> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED:
>> Shutdown
>> >> >> hook
>> >> >> > 2011-11-29 11:24:36,792 INFO
>> >> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs
>> >> shutdown
>> >> >> > hook thread.
>> >> >> > 2011-11-29 11:24:36,792 INFO
>> >> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>> >> >> finished.
>> >> >> >
>> >> >> > what is the cause for problem here, please help.
>> >> >> >
>> >> >> > On Mon, Nov 28, 2011 at 7:28 PM, Lars George <
>> lars.george@gmail.com>
>> >> >> wrote:
>> >> >> >
>> >> >> >> Hi,
>> >> >> >>
>> >> >> >> Did you add the list of servers to the regionservers file in the
>> >> >> >> $HBASE_HOME/conf/ dir? Are you using Cygwin? Or what else is your
>> >> >> >> environment?
>> >> >> >>
>> >> >> >> Lars
>> >> >> >>
>> >> >> >> On Nov 26, 2011, at 7:37 AM, Vamshi Krishna wrote:
>> >> >> >>
>> >> >> >> > Hi i am running hbase on 3 machines, on one node master and
>> >> >> regionserver,
>> >> >> >> > on other two nodes regionservers. i ran bin/start-hbase.sh on
>> >> master,
>> >> >> all
>> >> >> >> > the hbase daemons running on master node(HMaster, HQuorumpeer,
>> >> >> >> > HRegionServer), But on the other regionserver machines i could
>> not
>> >> >> find
>> >> >> >> any
>> >> >> >> > hbase daemons running. According to my knowledge the
>> HRegionserver
>> >> >> >> daemons
>> >> >> >> > should run on other machines automatically when i run
>> >> >> bin/start-hbase.sh
>> >> >> >> on
>> >> >> >> > master node, right? But its not happening so.
>> >> >> >> > More over its printing errors on my master node when i run
>> >> >> >> > bin/start-hbase.sh
>> >> >> >> >
>> >> >> >> > hbase-regionserver1: bash: line 0: cd: {HBASE_HOME}/bin/..: No
>> such
>> >> >> file
>> >> >> >> or
>> >> >> >> > directory
>> >> >> >> > hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No
>> >> such
>> >> >> file
>> >> >> >> > or directory
>> >> >> >> > hbase-regionserver2: bash: line 0: cd: {HBASE_HOME}/bin/..: No
>> such
>> >> >> file
>> >> >> >> or
>> >> >> >> > directory
>> >> >> >> > hbase-regionserver2: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No
>> >> such
>> >> >> file
>> >> >> >> > or directory ,
>> >> >> >> >
>> >> >> >> > but i could find out {HBASE_HOME}/bin/hbase-daemon.sh on both
>> >> >> machines.
>> >> >> >> > Infact the path of the {HBASE_HOME} folder on each of the
>> >> respective
>> >> >> >> > machines machines is same. i.e
>> >> >> >> > /home/hduser/Documents/HBASE_SOFTWRAE/hbase-0.90.4
>> >> >> >> >
>> >> >> >> > please can anybody help?
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > --
>> >> >> >> > *Regards*
>> >> >> >> > *
>> >> >> >> > Vamshi Krishna
>> >> >> >> > *
>> >> >> >>
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > *Regards*
>> >> >> > *
>> >> >> > Vamshi Krishna
>> >> >> > *
>> >> >> >
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > *Regards*
>> >> > *
>> >> > Vamshi Krishna
>> >> > *
>> >>
>> >
>> >
>> >
>> > --
>> > *Regards*
>> > *
>> > Vamshi Krishna
>> > *
>>
>
>
>
> --
> *Regards*
> *
> Vamshi Krishna
> *
Re: hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No
such file or directory
Posted by Vamshi Krishna <va...@gmail.com>.
I disabled the firewall in all the machines.Then i started
hbase(bin/start-hbase.sh), after 2-3 minutesi stopped hbase on master node
(bin/stop-hbase.sh). Even then HRegionserver daemon is running on region
server machine. So i checked regionserver machines log, the full log is
here
http://pastebin.com/FXsNLMD9
I found that regionserver machine is not able to communicate with master
though it is attempting to connect to master (hbase-master:60000)for every
one minute or so.
so i tried to do the following as you have told.
hduser@vamshi-laptop:~$ telnet hbase-master 60000
Trying 10.0.1.54...
telnet: Unable to connect to remote host: Connection refused
hduser@vamshi-laptop:~$ telnet hbase-master:60000
telnet: could not resolve hbase-master:60000/telnet: Name or service not
known
i made hbase-master to bind to 10.0.1.54 in /etc/hosts file, so i tried
even following also.
hduser@vamshi-laptop:~$ telnet 10.0.1.54 60000
Trying 10.0.1.54...
telnet: Unable to connect to remote host: Connection refused
hduser@vamshi-laptop:~$
So, i thought whether hbase-master is bound to port 60000 on the master
node or not, so while hbase master is running, i checked by typing
netstat -anp | grep 60000
i could not find 10.0.1.54:60000 any where on the console, rather i found
127.0.1.1:60000 in some places.
Is this the correct way of representing /etc/hosts file on master node?
127.0.0.1 localhost
127.0.1.1 vamshikrishna-laptop
10.0.1.54 hbase-master hadoop-namenode vamshikrishna-desktop
10.0.1.55 hbase-regionserver1 hadoop-datanode1
10.0.1.97 hbase-regionserver2 hadoop-datanode2 arun-desktop
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
Even i checked by typing the command on both master node and region server
machine
$ nmap -p 60000 hbase-master
then i got some thing on console like
60000/tcp closed
Please help me, i dont understand what went wrong in my set up.
On Thu, Dec 1, 2011 at 11:28 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> So since I don't see the rest of the log I'll have to assume that the
> region server was never able to connect to the master. Connection
> refused could be a firewall, start the master and then try to telnet
> from the other machines to master:60000.
>
> J-D
>
> On Thu, Dec 1, 2011 at 6:45 AM, Vamshi Krishna <va...@gmail.com>
> wrote:
> > I found in the logs of region server machines, i found this error (on
> both
> > regionserver machines)
> >
> > 2011-11-30 14:43:42,447 INFO org.apache.hadoop.ipc.HbaseRPC: Server at
> > hbase-master/10.0.1.54:60020 could not be reached after 1 tries, giving
> up.
> > *2011-11-30 14:44:37,762* WARN
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to
> > master. Retrying. Error was:
> > java.net.ConnectException: Connection refused
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> > at
> >
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
> > at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> > at $Proxy5.getProtocolVersion(Unknown Source)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
> > at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1462)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
> > at java.lang.Thread.run(Thread.java:662)
> > 2011-11-30 14:44:40,768 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to
> > Master server at hbase-master:60000
> > *2011-11-30 14:45:40,847* WARN
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to
> > master. Retrying. Error was:
> > java.net.ConnectException: Connection refused
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> > at
> >
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
> > at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> > at $Proxy5.getProtocolVersion(Unknown Source)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
> > at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1462)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
> > at java.lang.Thread.run(Thread.java:662)
> >
> >
> > and the same error is observed in the whole log repeatedly. After seeing
> it
> > what i understand is that some how master started HRegionServer daemons
> on
> > the machines but from then onwards the RegionServer machines are not able
> > to communicate with master. If we observe it is trying to communicate
> with
> > master for evry one minute.
> >
> > But i am not understanding where to check and modify the things.. please
> > help. i feel all connections are OK.
> >
> > On Thu, Dec 1, 2011 at 12:28 AM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
> >
> >> stop-hbase.sh only tells the master to stop, which in turn will tell
> >> the region servers to stop. If they are still running, it might be
> >> because of an error. Look at their logs to figure what's going on.
> >>
> >> J-D
> >>
> >> On Tue, Nov 29, 2011 at 10:46 PM, Vamshi Krishna <va...@gmail.com>
> >> wrote:
> >> > hey soryy for posting multiple times.
> >> > J-D, As you said, i refered to my regionserver log, there i found
> >> > Could not resolve the DNS name of vamshikrishna-desktop
> >> > so i added an alias ' vamshikrishna-desktop ' to its corresponding IP
> >> > address in /etc/hosts. So, from then master is able to run
> HRegionServer
> >> > daemon in the regionserver machines also.
> >> >
> >> > But the ONLY problem now is when i stop hbase on my master node by
> >> running
> >> > bin/stop-hbase.sh, all hbase daemons are stopping on matser node but
> NOT
> >> on
> >> > regionserver nodes.The HRegionServer daemon is still running on the
> other
> >> > regionserver machines.
> >> > I think the HRegionServer daemons on all the regionserver machines
> should
> >> > also get stopped, when we execute bin/sto-hbase.sh on master node,
> right?
> >> > please help..
> >> >
> >> > On Wed, Nov 30, 2011 at 12:38 AM, Jean-Daniel Cryans <
> >> jdcryans@apache.org>wrote:
> >> >
> >> >> You posted this in two threads, please refrain from doing this in the
> >> >> future.
> >> >>
> >> >> From what I can read in there, it tried to get the master address
> >> >> that's supposed to be in zookeeper but it failed because it was
> >> >> missing and then died. The way it's handled is a bit ugly but the
> >> >> effect is the same, take a look at the master log to see what's going
> >> >> on.
> >> >>
> >> >> J-D
> >> >>
> >> >> On Mon, Nov 28, 2011 at 10:33 PM, Vamshi Krishna <
> vamshi2105@gmail.com>
> >> >> wrote:
> >> >> > Hi Lars,
> >> >> > i am not using cygwin, i am using 3 ubuntu-10.04 machines.
> >> >> > Finally that problem i mentioned got resolved i.e now i can see the
> >> >> > following after i run bin/start-hbase.sh on my master machine,
> >> >> >
> >> >> > hbase-master: starting zookeeper, logging to
> >> >> >
> >> >>
> >>
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-zookeeper-vamshikrishna-desktop.out
> >> >> > starting master, logging to
> >> >> >
> >> >>
> >>
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/logs/hbase-hduser-master-vamshikrishna-desktop.out
> >> >> > hbase-regionserver2: starting regionserver, logging to
> >> >> >
> >> >>
> >>
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.out
> >> >> > hbase-regionserver1: starting regionserver, logging to
> >> >> >
> >> >>
> >>
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.out
> >> >> >
> >> >> > hbase-master: starting regionserver, logging to
> >> >> >
> >> >>
> >>
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshikrishna-desktop.out
> >> >> >
> >> >> >
> >> >> >
> >> >> > But still , if i do jps on both region server machines,
> HRegionServer
> >> >> > daemon is not seen.And i checked the logs on both the regionservere
> >> >> > machines, i.e
> >> >> >
> >> >> >
> >> >>
> >>
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.log
> >> >> > ---(hbase-regionserver1)
> >> >> > and
> >> >> >
> >> >>
> >>
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.log
> >> >> > ---(hbase-regionserver2)
> >> >> >
> >> >> > i found the same in both .it is like this, (here
> >> vamshikrishna-desktop
> >> >> is
> >> >> > the master and namenode machine)
> >> >> >
> >> >> >
> >> >> >
> >> >> > 2011-11-29 11:24:36,714 ERROR
> org.apache.hadoop.hbase.HServerAddress:
> >> >> Could
> >> >> > not resolve the DNS name of vamshikrishna-desktop
> >> >> > 2011-11-29 11:24:36,716 FATAL
> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region
> >> >> server
> >> >> > serverName=vamshi-laptop,60020,1322546076276, load=(requests=0,
> >> >> regions=0,
> >> >> > usedHeap=0, maxHeap=0): Unhandled exception: hostname can't be null
> >> >> > java.lang.IllegalArgumentException: hostname can't be null
> >> >> > at java.net.InetSocketAddress.<init>(InetSocketAddress.java:121)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.hadoop.hbase.HServerAddress.getResolvedAddress(HServerAddress.java:108)
> >> >> > at
> >> >> org.apache.hadoop.hbase.HServerAddress.<init>(HServerAddress.java:64)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.hadoop.hbase.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:63)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMasterAddress(HRegionServer.java:1484)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1457)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
> >> >> > at java.lang.Thread.run(Thread.java:662)
> >> >> > 2011-11-29 11:24:36,719 INFO
> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED:
> Unhandled
> >> >> > exception: hostname can't be null
> >> >> > 2011-11-29 11:24:36,719 INFO org.apache.hadoop.ipc.HBaseServer:
> >> Stopping
> >> >> > server on 60020
> >> >> > 2011-11-29 11:24:36,749 INFO
> >> >> > org.apache.hadoop.hbase.regionserver.StoreFile: Allocating
> >> LruBlockCache
> >> >> > with maximum size 199.4m
> >> >> > 2011-11-29 11:24:36,775 INFO
> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server
> >> at:
> >> >> > vamshi-laptop,60020,1322546076276
> >> >> > 2011-11-29 11:24:36,775 DEBUG
> >> >> > org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog
> >> tracker
> >> >> > org.apache.hadoop.hbase.catalog.CatalogTracker@da2cef
> >> >> > 2011-11-29 11:24:36,776 INFO
> >> org.apache.hadoop.hbase.regionserver.Leases:
> >> >> > regionserver60020 closing leases
> >> >> > 2011-11-29 11:24:36,776 INFO
> >> org.apache.hadoop.hbase.regionserver.Leases:
> >> >> > regionserver60020 closed leases
> >> >> > 2011-11-29 11:24:36,776 INFO
> >> >> >
> >> >>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> >> >> > Closed zookeeper sessionid=0x133ede31f3c0004
> >> >> > 2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ClientCnxn:
> >> EventThread
> >> >> > shut down
> >> >> > 2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ZooKeeper:
> Session:
> >> >> > 0x133ede31f3c0004 closed
> >> >> > 2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ClientCnxn:
> >> EventThread
> >> >> > shut down
> >> >> > 2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ZooKeeper:
> Session:
> >> >> > 0x133ede31f3c0005 closed
> >> >> > 2011-11-29 11:24:36,792 INFO
> >> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
> >> >> starting;
> >> >> > hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-14,5,main]
> >> >> > 2011-11-29 11:24:36,792 INFO
> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED:
> Shutdown
> >> >> hook
> >> >> > 2011-11-29 11:24:36,792 INFO
> >> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs
> >> shutdown
> >> >> > hook thread.
> >> >> > 2011-11-29 11:24:36,792 INFO
> >> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
> >> >> finished.
> >> >> >
> >> >> > what is the cause for problem here, please help.
> >> >> >
> >> >> > On Mon, Nov 28, 2011 at 7:28 PM, Lars George <
> lars.george@gmail.com>
> >> >> wrote:
> >> >> >
> >> >> >> Hi,
> >> >> >>
> >> >> >> Did you add the list of servers to the regionservers file in the
> >> >> >> $HBASE_HOME/conf/ dir? Are you using Cygwin? Or what else is your
> >> >> >> environment?
> >> >> >>
> >> >> >> Lars
> >> >> >>
> >> >> >> On Nov 26, 2011, at 7:37 AM, Vamshi Krishna wrote:
> >> >> >>
> >> >> >> > Hi i am running hbase on 3 machines, on one node master and
> >> >> regionserver,
> >> >> >> > on other two nodes regionservers. i ran bin/start-hbase.sh on
> >> master,
> >> >> all
> >> >> >> > the hbase daemons running on master node(HMaster, HQuorumpeer,
> >> >> >> > HRegionServer), But on the other regionserver machines i could
> not
> >> >> find
> >> >> >> any
> >> >> >> > hbase daemons running. According to my knowledge the
> HRegionserver
> >> >> >> daemons
> >> >> >> > should run on other machines automatically when i run
> >> >> bin/start-hbase.sh
> >> >> >> on
> >> >> >> > master node, right? But its not happening so.
> >> >> >> > More over its printing errors on my master node when i run
> >> >> >> > bin/start-hbase.sh
> >> >> >> >
> >> >> >> > hbase-regionserver1: bash: line 0: cd: {HBASE_HOME}/bin/..: No
> such
> >> >> file
> >> >> >> or
> >> >> >> > directory
> >> >> >> > hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No
> >> such
> >> >> file
> >> >> >> > or directory
> >> >> >> > hbase-regionserver2: bash: line 0: cd: {HBASE_HOME}/bin/..: No
> such
> >> >> file
> >> >> >> or
> >> >> >> > directory
> >> >> >> > hbase-regionserver2: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No
> >> such
> >> >> file
> >> >> >> > or directory ,
> >> >> >> >
> >> >> >> > but i could find out {HBASE_HOME}/bin/hbase-daemon.sh on both
> >> >> machines.
> >> >> >> > Infact the path of the {HBASE_HOME} folder on each of the
> >> respective
> >> >> >> > machines machines is same. i.e
> >> >> >> > /home/hduser/Documents/HBASE_SOFTWRAE/hbase-0.90.4
> >> >> >> >
> >> >> >> > please can anybody help?
> >> >> >> >
> >> >> >> >
> >> >> >> > --
> >> >> >> > *Regards*
> >> >> >> > *
> >> >> >> > Vamshi Krishna
> >> >> >> > *
> >> >> >>
> >> >> >>
> >> >> >
> >> >> >
> >> >> > --
> >> >> > *Regards*
> >> >> > *
> >> >> > Vamshi Krishna
> >> >> > *
> >> >> >
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > *Regards*
> >> > *
> >> > Vamshi Krishna
> >> > *
> >>
> >
> >
> >
> > --
> > *Regards*
> > *
> > Vamshi Krishna
> > *
>
--
*Regards*
*
Vamshi Krishna
*
Re: hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No
such file or directory
Posted by Jean-Daniel Cryans <jd...@apache.org>.
So since I don't see the rest of the log I'll have to assume that the
region server was never able to connect to the master. Connection
refused could be a firewall, start the master and then try to telnet
from the other machines to master:60000.
J-D
On Thu, Dec 1, 2011 at 6:45 AM, Vamshi Krishna <va...@gmail.com> wrote:
> I found in the logs of region server machines, i found this error (on both
> regionserver machines)
>
> 2011-11-30 14:43:42,447 INFO org.apache.hadoop.ipc.HbaseRPC: Server at
> hbase-master/10.0.1.54:60020 could not be reached after 1 tries, giving up.
> *2011-11-30 14:44:37,762* WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to
> master. Retrying. Error was:
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> at $Proxy5.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1462)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
> at java.lang.Thread.run(Thread.java:662)
> 2011-11-30 14:44:40,768 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to
> Master server at hbase-master:60000
> *2011-11-30 14:45:40,847* WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to
> master. Retrying. Error was:
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> at $Proxy5.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1462)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
> at java.lang.Thread.run(Thread.java:662)
>
>
> and the same error is observed in the whole log repeatedly. After seeing it
> what i understand is that some how master started HRegionServer daemons on
> the machines but from then onwards the RegionServer machines are not able
> to communicate with master. If we observe it is trying to communicate with
> master for evry one minute.
>
> But i am not understanding where to check and modify the things.. please
> help. i feel all connections are OK.
>
> On Thu, Dec 1, 2011 at 12:28 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> stop-hbase.sh only tells the master to stop, which in turn will tell
>> the region servers to stop. If they are still running, it might be
>> because of an error. Look at their logs to figure what's going on.
>>
>> J-D
>>
>> On Tue, Nov 29, 2011 at 10:46 PM, Vamshi Krishna <va...@gmail.com>
>> wrote:
>> > hey soryy for posting multiple times.
>> > J-D, As you said, i refered to my regionserver log, there i found
>> > Could not resolve the DNS name of vamshikrishna-desktop
>> > so i added an alias ' vamshikrishna-desktop ' to its corresponding IP
>> > address in /etc/hosts. So, from then master is able to run HRegionServer
>> > daemon in the regionserver machines also.
>> >
>> > But the ONLY problem now is when i stop hbase on my master node by
>> running
>> > bin/stop-hbase.sh, all hbase daemons are stopping on matser node but NOT
>> on
>> > regionserver nodes.The HRegionServer daemon is still running on the other
>> > regionserver machines.
>> > I think the HRegionServer daemons on all the regionserver machines should
>> > also get stopped, when we execute bin/sto-hbase.sh on master node, right?
>> > please help..
>> >
>> > On Wed, Nov 30, 2011 at 12:38 AM, Jean-Daniel Cryans <
>> jdcryans@apache.org>wrote:
>> >
>> >> You posted this in two threads, please refrain from doing this in the
>> >> future.
>> >>
>> >> From what I can read in there, it tried to get the master address
>> >> that's supposed to be in zookeeper but it failed because it was
>> >> missing and then died. The way it's handled is a bit ugly but the
>> >> effect is the same, take a look at the master log to see what's going
>> >> on.
>> >>
>> >> J-D
>> >>
>> >> On Mon, Nov 28, 2011 at 10:33 PM, Vamshi Krishna <va...@gmail.com>
>> >> wrote:
>> >> > Hi Lars,
>> >> > i am not using cygwin, i am using 3 ubuntu-10.04 machines.
>> >> > Finally that problem i mentioned got resolved i.e now i can see the
>> >> > following after i run bin/start-hbase.sh on my master machine,
>> >> >
>> >> > hbase-master: starting zookeeper, logging to
>> >> >
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-zookeeper-vamshikrishna-desktop.out
>> >> > starting master, logging to
>> >> >
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/logs/hbase-hduser-master-vamshikrishna-desktop.out
>> >> > hbase-regionserver2: starting regionserver, logging to
>> >> >
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.out
>> >> > hbase-regionserver1: starting regionserver, logging to
>> >> >
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.out
>> >> >
>> >> > hbase-master: starting regionserver, logging to
>> >> >
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshikrishna-desktop.out
>> >> >
>> >> >
>> >> >
>> >> > But still , if i do jps on both region server machines, HRegionServer
>> >> > daemon is not seen.And i checked the logs on both the regionservere
>> >> > machines, i.e
>> >> >
>> >> >
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.log
>> >> > ---(hbase-regionserver1)
>> >> > and
>> >> >
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.log
>> >> > ---(hbase-regionserver2)
>> >> >
>> >> > i found the same in both .it is like this, (here
>> vamshikrishna-desktop
>> >> is
>> >> > the master and namenode machine)
>> >> >
>> >> >
>> >> >
>> >> > 2011-11-29 11:24:36,714 ERROR org.apache.hadoop.hbase.HServerAddress:
>> >> Could
>> >> > not resolve the DNS name of vamshikrishna-desktop
>> >> > 2011-11-29 11:24:36,716 FATAL
>> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region
>> >> server
>> >> > serverName=vamshi-laptop,60020,1322546076276, load=(requests=0,
>> >> regions=0,
>> >> > usedHeap=0, maxHeap=0): Unhandled exception: hostname can't be null
>> >> > java.lang.IllegalArgumentException: hostname can't be null
>> >> > at java.net.InetSocketAddress.<init>(InetSocketAddress.java:121)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.HServerAddress.getResolvedAddress(HServerAddress.java:108)
>> >> > at
>> >> org.apache.hadoop.hbase.HServerAddress.<init>(HServerAddress.java:64)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:63)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getMasterAddress(HRegionServer.java:1484)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1457)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
>> >> > at java.lang.Thread.run(Thread.java:662)
>> >> > 2011-11-29 11:24:36,719 INFO
>> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unhandled
>> >> > exception: hostname can't be null
>> >> > 2011-11-29 11:24:36,719 INFO org.apache.hadoop.ipc.HBaseServer:
>> Stopping
>> >> > server on 60020
>> >> > 2011-11-29 11:24:36,749 INFO
>> >> > org.apache.hadoop.hbase.regionserver.StoreFile: Allocating
>> LruBlockCache
>> >> > with maximum size 199.4m
>> >> > 2011-11-29 11:24:36,775 INFO
>> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server
>> at:
>> >> > vamshi-laptop,60020,1322546076276
>> >> > 2011-11-29 11:24:36,775 DEBUG
>> >> > org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog
>> tracker
>> >> > org.apache.hadoop.hbase.catalog.CatalogTracker@da2cef
>> >> > 2011-11-29 11:24:36,776 INFO
>> org.apache.hadoop.hbase.regionserver.Leases:
>> >> > regionserver60020 closing leases
>> >> > 2011-11-29 11:24:36,776 INFO
>> org.apache.hadoop.hbase.regionserver.Leases:
>> >> > regionserver60020 closed leases
>> >> > 2011-11-29 11:24:36,776 INFO
>> >> >
>> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
>> >> > Closed zookeeper sessionid=0x133ede31f3c0004
>> >> > 2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ClientCnxn:
>> EventThread
>> >> > shut down
>> >> > 2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ZooKeeper: Session:
>> >> > 0x133ede31f3c0004 closed
>> >> > 2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ClientCnxn:
>> EventThread
>> >> > shut down
>> >> > 2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ZooKeeper: Session:
>> >> > 0x133ede31f3c0005 closed
>> >> > 2011-11-29 11:24:36,792 INFO
>> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>> >> starting;
>> >> > hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-14,5,main]
>> >> > 2011-11-29 11:24:36,792 INFO
>> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
>> >> hook
>> >> > 2011-11-29 11:24:36,792 INFO
>> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs
>> shutdown
>> >> > hook thread.
>> >> > 2011-11-29 11:24:36,792 INFO
>> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>> >> finished.
>> >> >
>> >> > what is the cause for problem here, please help.
>> >> >
>> >> > On Mon, Nov 28, 2011 at 7:28 PM, Lars George <la...@gmail.com>
>> >> wrote:
>> >> >
>> >> >> Hi,
>> >> >>
>> >> >> Did you add the list of servers to the regionservers file in the
>> >> >> $HBASE_HOME/conf/ dir? Are you using Cygwin? Or what else is your
>> >> >> environment?
>> >> >>
>> >> >> Lars
>> >> >>
>> >> >> On Nov 26, 2011, at 7:37 AM, Vamshi Krishna wrote:
>> >> >>
>> >> >> > Hi i am running hbase on 3 machines, on one node master and
>> >> regionserver,
>> >> >> > on other two nodes regionservers. i ran bin/start-hbase.sh on
>> master,
>> >> all
>> >> >> > the hbase daemons running on master node(HMaster, HQuorumpeer,
>> >> >> > HRegionServer), But on the other regionserver machines i could not
>> >> find
>> >> >> any
>> >> >> > hbase daemons running. According to my knowledge the HRegionserver
>> >> >> daemons
>> >> >> > should run on other machines automatically when i run
>> >> bin/start-hbase.sh
>> >> >> on
>> >> >> > master node, right? But its not happening so.
>> >> >> > More over its printing errors on my master node when i run
>> >> >> > bin/start-hbase.sh
>> >> >> >
>> >> >> > hbase-regionserver1: bash: line 0: cd: {HBASE_HOME}/bin/..: No such
>> >> file
>> >> >> or
>> >> >> > directory
>> >> >> > hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No
>> such
>> >> file
>> >> >> > or directory
>> >> >> > hbase-regionserver2: bash: line 0: cd: {HBASE_HOME}/bin/..: No such
>> >> file
>> >> >> or
>> >> >> > directory
>> >> >> > hbase-regionserver2: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No
>> such
>> >> file
>> >> >> > or directory ,
>> >> >> >
>> >> >> > but i could find out {HBASE_HOME}/bin/hbase-daemon.sh on both
>> >> machines.
>> >> >> > Infact the path of the {HBASE_HOME} folder on each of the
>> respective
>> >> >> > machines machines is same. i.e
>> >> >> > /home/hduser/Documents/HBASE_SOFTWRAE/hbase-0.90.4
>> >> >> >
>> >> >> > please can anybody help?
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > *Regards*
>> >> >> > *
>> >> >> > Vamshi Krishna
>> >> >> > *
>> >> >>
>> >> >>
>> >> >
>> >> >
>> >> > --
>> >> > *Regards*
>> >> > *
>> >> > Vamshi Krishna
>> >> > *
>> >> >
>> >>
>> >
>> >
>> >
>> > --
>> > *Regards*
>> > *
>> > Vamshi Krishna
>> > *
>>
>
>
>
> --
> *Regards*
> *
> Vamshi Krishna
> *
Re: hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No
such file or directory
Posted by Vamshi Krishna <va...@gmail.com>.
I found in the logs of region server machines, i found this error (on both
regionserver machines)
2011-11-30 14:43:42,447 INFO org.apache.hadoop.ipc.HbaseRPC: Server at
hbase-master/10.0.1.54:60020 could not be reached after 1 tries, giving up.
*2011-11-30 14:44:37,762* WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to
master. Retrying. Error was:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at
org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy5.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1462)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
at java.lang.Thread.run(Thread.java:662)
2011-11-30 14:44:40,768 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to
Master server at hbase-master:60000
*2011-11-30 14:45:40,847* WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to
master. Retrying. Error was:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at
org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy5.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1462)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
at java.lang.Thread.run(Thread.java:662)
and the same error is observed in the whole log repeatedly. After seeing it
what i understand is that some how master started HRegionServer daemons on
the machines but from then onwards the RegionServer machines are not able
to communicate with master. If we observe it is trying to communicate with
master for evry one minute.
But i am not understanding where to check and modify the things.. please
help. i feel all connections are OK.
On Thu, Dec 1, 2011 at 12:28 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> stop-hbase.sh only tells the master to stop, which in turn will tell
> the region servers to stop. If they are still running, it might be
> because of an error. Look at their logs to figure what's going on.
>
> J-D
>
> On Tue, Nov 29, 2011 at 10:46 PM, Vamshi Krishna <va...@gmail.com>
> wrote:
> > hey soryy for posting multiple times.
> > J-D, As you said, i refered to my regionserver log, there i found
> > Could not resolve the DNS name of vamshikrishna-desktop
> > so i added an alias ' vamshikrishna-desktop ' to its corresponding IP
> > address in /etc/hosts. So, from then master is able to run HRegionServer
> > daemon in the regionserver machines also.
> >
> > But the ONLY problem now is when i stop hbase on my master node by
> running
> > bin/stop-hbase.sh, all hbase daemons are stopping on matser node but NOT
> on
> > regionserver nodes.The HRegionServer daemon is still running on the other
> > regionserver machines.
> > I think the HRegionServer daemons on all the regionserver machines should
> > also get stopped, when we execute bin/sto-hbase.sh on master node, right?
> > please help..
> >
> > On Wed, Nov 30, 2011 at 12:38 AM, Jean-Daniel Cryans <
> jdcryans@apache.org>wrote:
> >
> >> You posted this in two threads, please refrain from doing this in the
> >> future.
> >>
> >> From what I can read in there, it tried to get the master address
> >> that's supposed to be in zookeeper but it failed because it was
> >> missing and then died. The way it's handled is a bit ugly but the
> >> effect is the same, take a look at the master log to see what's going
> >> on.
> >>
> >> J-D
> >>
> >> On Mon, Nov 28, 2011 at 10:33 PM, Vamshi Krishna <va...@gmail.com>
> >> wrote:
> >> > Hi Lars,
> >> > i am not using cygwin, i am using 3 ubuntu-10.04 machines.
> >> > Finally that problem i mentioned got resolved i.e now i can see the
> >> > following after i run bin/start-hbase.sh on my master machine,
> >> >
> >> > hbase-master: starting zookeeper, logging to
> >> >
> >>
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-zookeeper-vamshikrishna-desktop.out
> >> > starting master, logging to
> >> >
> >>
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/logs/hbase-hduser-master-vamshikrishna-desktop.out
> >> > hbase-regionserver2: starting regionserver, logging to
> >> >
> >>
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.out
> >> > hbase-regionserver1: starting regionserver, logging to
> >> >
> >>
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.out
> >> >
> >> > hbase-master: starting regionserver, logging to
> >> >
> >>
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshikrishna-desktop.out
> >> >
> >> >
> >> >
> >> > But still , if i do jps on both region server machines, HRegionServer
> >> > daemon is not seen.And i checked the logs on both the regionservere
> >> > machines, i.e
> >> >
> >> >
> >>
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.log
> >> > ---(hbase-regionserver1)
> >> > and
> >> >
> >>
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.log
> >> > ---(hbase-regionserver2)
> >> >
> >> > i found the same in both .it is like this, (here
> vamshikrishna-desktop
> >> is
> >> > the master and namenode machine)
> >> >
> >> >
> >> >
> >> > 2011-11-29 11:24:36,714 ERROR org.apache.hadoop.hbase.HServerAddress:
> >> Could
> >> > not resolve the DNS name of vamshikrishna-desktop
> >> > 2011-11-29 11:24:36,716 FATAL
> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region
> >> server
> >> > serverName=vamshi-laptop,60020,1322546076276, load=(requests=0,
> >> regions=0,
> >> > usedHeap=0, maxHeap=0): Unhandled exception: hostname can't be null
> >> > java.lang.IllegalArgumentException: hostname can't be null
> >> > at java.net.InetSocketAddress.<init>(InetSocketAddress.java:121)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.HServerAddress.getResolvedAddress(HServerAddress.java:108)
> >> > at
> >> org.apache.hadoop.hbase.HServerAddress.<init>(HServerAddress.java:64)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:63)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMasterAddress(HRegionServer.java:1484)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1457)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
> >> > at java.lang.Thread.run(Thread.java:662)
> >> > 2011-11-29 11:24:36,719 INFO
> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unhandled
> >> > exception: hostname can't be null
> >> > 2011-11-29 11:24:36,719 INFO org.apache.hadoop.ipc.HBaseServer:
> Stopping
> >> > server on 60020
> >> > 2011-11-29 11:24:36,749 INFO
> >> > org.apache.hadoop.hbase.regionserver.StoreFile: Allocating
> LruBlockCache
> >> > with maximum size 199.4m
> >> > 2011-11-29 11:24:36,775 INFO
> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server
> at:
> >> > vamshi-laptop,60020,1322546076276
> >> > 2011-11-29 11:24:36,775 DEBUG
> >> > org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog
> tracker
> >> > org.apache.hadoop.hbase.catalog.CatalogTracker@da2cef
> >> > 2011-11-29 11:24:36,776 INFO
> org.apache.hadoop.hbase.regionserver.Leases:
> >> > regionserver60020 closing leases
> >> > 2011-11-29 11:24:36,776 INFO
> org.apache.hadoop.hbase.regionserver.Leases:
> >> > regionserver60020 closed leases
> >> > 2011-11-29 11:24:36,776 INFO
> >> >
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> >> > Closed zookeeper sessionid=0x133ede31f3c0004
> >> > 2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ClientCnxn:
> EventThread
> >> > shut down
> >> > 2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ZooKeeper: Session:
> >> > 0x133ede31f3c0004 closed
> >> > 2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ClientCnxn:
> EventThread
> >> > shut down
> >> > 2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ZooKeeper: Session:
> >> > 0x133ede31f3c0005 closed
> >> > 2011-11-29 11:24:36,792 INFO
> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
> >> starting;
> >> > hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-14,5,main]
> >> > 2011-11-29 11:24:36,792 INFO
> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
> >> hook
> >> > 2011-11-29 11:24:36,792 INFO
> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs
> shutdown
> >> > hook thread.
> >> > 2011-11-29 11:24:36,792 INFO
> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
> >> finished.
> >> >
> >> > what is the cause for problem here, please help.
> >> >
> >> > On Mon, Nov 28, 2011 at 7:28 PM, Lars George <la...@gmail.com>
> >> wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> Did you add the list of servers to the regionservers file in the
> >> >> $HBASE_HOME/conf/ dir? Are you using Cygwin? Or what else is your
> >> >> environment?
> >> >>
> >> >> Lars
> >> >>
> >> >> On Nov 26, 2011, at 7:37 AM, Vamshi Krishna wrote:
> >> >>
> >> >> > Hi i am running hbase on 3 machines, on one node master and
> >> regionserver,
> >> >> > on other two nodes regionservers. i ran bin/start-hbase.sh on
> master,
> >> all
> >> >> > the hbase daemons running on master node(HMaster, HQuorumpeer,
> >> >> > HRegionServer), But on the other regionserver machines i could not
> >> find
> >> >> any
> >> >> > hbase daemons running. According to my knowledge the HRegionserver
> >> >> daemons
> >> >> > should run on other machines automatically when i run
> >> bin/start-hbase.sh
> >> >> on
> >> >> > master node, right? But its not happening so.
> >> >> > More over its printing errors on my master node when i run
> >> >> > bin/start-hbase.sh
> >> >> >
> >> >> > hbase-regionserver1: bash: line 0: cd: {HBASE_HOME}/bin/..: No such
> >> file
> >> >> or
> >> >> > directory
> >> >> > hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No
> such
> >> file
> >> >> > or directory
> >> >> > hbase-regionserver2: bash: line 0: cd: {HBASE_HOME}/bin/..: No such
> >> file
> >> >> or
> >> >> > directory
> >> >> > hbase-regionserver2: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No
> such
> >> file
> >> >> > or directory ,
> >> >> >
> >> >> > but i could find out {HBASE_HOME}/bin/hbase-daemon.sh on both
> >> machines.
> >> >> > Infact the path of the {HBASE_HOME} folder on each of the
> respective
> >> >> > machines machines is same. i.e
> >> >> > /home/hduser/Documents/HBASE_SOFTWRAE/hbase-0.90.4
> >> >> >
> >> >> > please can anybody help?
> >> >> >
> >> >> >
> >> >> > --
> >> >> > *Regards*
> >> >> > *
> >> >> > Vamshi Krishna
> >> >> > *
> >> >>
> >> >>
> >> >
> >> >
> >> > --
> >> > *Regards*
> >> > *
> >> > Vamshi Krishna
> >> > *
> >> >
> >>
> >
> >
> >
> > --
> > *Regards*
> > *
> > Vamshi Krishna
> > *
>
--
*Regards*
*
Vamshi Krishna
*
Re: hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No
such file or directory
Posted by Jean-Daniel Cryans <jd...@apache.org>.
stop-hbase.sh only tells the master to stop, which in turn will tell
the region servers to stop. If they are still running, it might be
because of an error. Look at their logs to figure what's going on.
J-D
On Tue, Nov 29, 2011 at 10:46 PM, Vamshi Krishna <va...@gmail.com> wrote:
> hey soryy for posting multiple times.
> J-D, As you said, i refered to my regionserver log, there i found
> Could not resolve the DNS name of vamshikrishna-desktop
> so i added an alias ' vamshikrishna-desktop ' to its corresponding IP
> address in /etc/hosts. So, from then master is able to run HRegionServer
> daemon in the regionserver machines also.
>
> But the ONLY problem now is when i stop hbase on my master node by running
> bin/stop-hbase.sh, all hbase daemons are stopping on matser node but NOT on
> regionserver nodes.The HRegionServer daemon is still running on the other
> regionserver machines.
> I think the HRegionServer daemons on all the regionserver machines should
> also get stopped, when we execute bin/sto-hbase.sh on master node, right?
> please help..
>
> On Wed, Nov 30, 2011 at 12:38 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> You posted this in two threads, please refrain from doing this in the
>> future.
>>
>> From what I can read in there, it tried to get the master address
>> that's supposed to be in zookeeper but it failed because it was
>> missing and then died. The way it's handled is a bit ugly but the
>> effect is the same, take a look at the master log to see what's going
>> on.
>>
>> J-D
>>
>> On Mon, Nov 28, 2011 at 10:33 PM, Vamshi Krishna <va...@gmail.com>
>> wrote:
>> > Hi Lars,
>> > i am not using cygwin, i am using 3 ubuntu-10.04 machines.
>> > Finally that problem i mentioned got resolved i.e now i can see the
>> > following after i run bin/start-hbase.sh on my master machine,
>> >
>> > hbase-master: starting zookeeper, logging to
>> >
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-zookeeper-vamshikrishna-desktop.out
>> > starting master, logging to
>> >
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/logs/hbase-hduser-master-vamshikrishna-desktop.out
>> > hbase-regionserver2: starting regionserver, logging to
>> >
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.out
>> > hbase-regionserver1: starting regionserver, logging to
>> >
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.out
>> >
>> > hbase-master: starting regionserver, logging to
>> >
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshikrishna-desktop.out
>> >
>> >
>> >
>> > But still , if i do jps on both region server machines, HRegionServer
>> > daemon is not seen.And i checked the logs on both the regionservere
>> > machines, i.e
>> >
>> >
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.log
>> > ---(hbase-regionserver1)
>> > and
>> >
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.log
>> > ---(hbase-regionserver2)
>> >
>> > i found the same in both .it is like this, (here vamshikrishna-desktop
>> is
>> > the master and namenode machine)
>> >
>> >
>> >
>> > 2011-11-29 11:24:36,714 ERROR org.apache.hadoop.hbase.HServerAddress:
>> Could
>> > not resolve the DNS name of vamshikrishna-desktop
>> > 2011-11-29 11:24:36,716 FATAL
>> > org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region
>> server
>> > serverName=vamshi-laptop,60020,1322546076276, load=(requests=0,
>> regions=0,
>> > usedHeap=0, maxHeap=0): Unhandled exception: hostname can't be null
>> > java.lang.IllegalArgumentException: hostname can't be null
>> > at java.net.InetSocketAddress.<init>(InetSocketAddress.java:121)
>> > at
>> >
>> org.apache.hadoop.hbase.HServerAddress.getResolvedAddress(HServerAddress.java:108)
>> > at
>> org.apache.hadoop.hbase.HServerAddress.<init>(HServerAddress.java:64)
>> > at
>> >
>> org.apache.hadoop.hbase.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:63)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getMasterAddress(HRegionServer.java:1484)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1457)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
>> > at java.lang.Thread.run(Thread.java:662)
>> > 2011-11-29 11:24:36,719 INFO
>> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unhandled
>> > exception: hostname can't be null
>> > 2011-11-29 11:24:36,719 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>> > server on 60020
>> > 2011-11-29 11:24:36,749 INFO
>> > org.apache.hadoop.hbase.regionserver.StoreFile: Allocating LruBlockCache
>> > with maximum size 199.4m
>> > 2011-11-29 11:24:36,775 INFO
>> > org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at:
>> > vamshi-laptop,60020,1322546076276
>> > 2011-11-29 11:24:36,775 DEBUG
>> > org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker
>> > org.apache.hadoop.hbase.catalog.CatalogTracker@da2cef
>> > 2011-11-29 11:24:36,776 INFO org.apache.hadoop.hbase.regionserver.Leases:
>> > regionserver60020 closing leases
>> > 2011-11-29 11:24:36,776 INFO org.apache.hadoop.hbase.regionserver.Leases:
>> > regionserver60020 closed leases
>> > 2011-11-29 11:24:36,776 INFO
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
>> > Closed zookeeper sessionid=0x133ede31f3c0004
>> > 2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ClientCnxn: EventThread
>> > shut down
>> > 2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ZooKeeper: Session:
>> > 0x133ede31f3c0004 closed
>> > 2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ClientCnxn: EventThread
>> > shut down
>> > 2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ZooKeeper: Session:
>> > 0x133ede31f3c0005 closed
>> > 2011-11-29 11:24:36,792 INFO
>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>> starting;
>> > hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-14,5,main]
>> > 2011-11-29 11:24:36,792 INFO
>> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
>> hook
>> > 2011-11-29 11:24:36,792 INFO
>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs shutdown
>> > hook thread.
>> > 2011-11-29 11:24:36,792 INFO
>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>> finished.
>> >
>> > what is the cause for problem here, please help.
>> >
>> > On Mon, Nov 28, 2011 at 7:28 PM, Lars George <la...@gmail.com>
>> wrote:
>> >
>> >> Hi,
>> >>
>> >> Did you add the list of servers to the regionservers file in the
>> >> $HBASE_HOME/conf/ dir? Are you using Cygwin? Or what else is your
>> >> environment?
>> >>
>> >> Lars
>> >>
>> >> On Nov 26, 2011, at 7:37 AM, Vamshi Krishna wrote:
>> >>
>> >> > Hi i am running hbase on 3 machines, on one node master and
>> regionserver,
>> >> > on other two nodes regionservers. i ran bin/start-hbase.sh on master,
>> all
>> >> > the hbase daemons running on master node(HMaster, HQuorumpeer,
>> >> > HRegionServer), But on the other regionserver machines i could not
>> find
>> >> any
>> >> > hbase daemons running. According to my knowledge the HRegionserver
>> >> daemons
>> >> > should run on other machines automatically when i run
>> bin/start-hbase.sh
>> >> on
>> >> > master node, right? But its not happening so.
>> >> > More over its printing errors on my master node when i run
>> >> > bin/start-hbase.sh
>> >> >
>> >> > hbase-regionserver1: bash: line 0: cd: {HBASE_HOME}/bin/..: No such
>> file
>> >> or
>> >> > directory
>> >> > hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No such
>> file
>> >> > or directory
>> >> > hbase-regionserver2: bash: line 0: cd: {HBASE_HOME}/bin/..: No such
>> file
>> >> or
>> >> > directory
>> >> > hbase-regionserver2: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No such
>> file
>> >> > or directory ,
>> >> >
>> >> > but i could find out {HBASE_HOME}/bin/hbase-daemon.sh on both
>> machines.
>> >> > Infact the path of the {HBASE_HOME} folder on each of the respective
>> >> > machines machines is same. i.e
>> >> > /home/hduser/Documents/HBASE_SOFTWRAE/hbase-0.90.4
>> >> >
>> >> > please can anybody help?
>> >> >
>> >> >
>> >> > --
>> >> > *Regards*
>> >> > *
>> >> > Vamshi Krishna
>> >> > *
>> >>
>> >>
>> >
>> >
>> > --
>> > *Regards*
>> > *
>> > Vamshi Krishna
>> > *
>> >
>>
>
>
>
> --
> *Regards*
> *
> Vamshi Krishna
> *
Re: hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No
such file or directory
Posted by Vamshi Krishna <va...@gmail.com>.
hey soryy for posting multiple times.
J-D, As you said, i refered to my regionserver log, there i found
Could not resolve the DNS name of vamshikrishna-desktop
so i added an alias ' vamshikrishna-desktop ' to its corresponding IP
address in /etc/hosts. So, from then master is able to run HRegionServer
daemon in the regionserver machines also.
But the ONLY problem now is when i stop hbase on my master node by running
bin/stop-hbase.sh, all hbase daemons are stopping on matser node but NOT on
regionserver nodes.The HRegionServer daemon is still running on the other
regionserver machines.
I think the HRegionServer daemons on all the regionserver machines should
also get stopped, when we execute bin/sto-hbase.sh on master node, right?
please help..
On Wed, Nov 30, 2011 at 12:38 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> You posted this in two threads, please refrain from doing this in the
> future.
>
> From what I can read in there, it tried to get the master address
> that's supposed to be in zookeeper but it failed because it was
> missing and then died. The way it's handled is a bit ugly but the
> effect is the same, take a look at the master log to see what's going
> on.
>
> J-D
>
> On Mon, Nov 28, 2011 at 10:33 PM, Vamshi Krishna <va...@gmail.com>
> wrote:
> > Hi Lars,
> > i am not using cygwin, i am using 3 ubuntu-10.04 machines.
> > Finally that problem i mentioned got resolved i.e now i can see the
> > following after i run bin/start-hbase.sh on my master machine,
> >
> > hbase-master: starting zookeeper, logging to
> >
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-zookeeper-vamshikrishna-desktop.out
> > starting master, logging to
> >
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/logs/hbase-hduser-master-vamshikrishna-desktop.out
> > hbase-regionserver2: starting regionserver, logging to
> >
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.out
> > hbase-regionserver1: starting regionserver, logging to
> >
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.out
> >
> > hbase-master: starting regionserver, logging to
> >
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshikrishna-desktop.out
> >
> >
> >
> > But still , if i do jps on both region server machines, HRegionServer
> > daemon is not seen.And i checked the logs on both the regionservere
> > machines, i.e
> >
> >
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.log
> > ---(hbase-regionserver1)
> > and
> >
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.log
> > ---(hbase-regionserver2)
> >
> > i found the same in both .it is like this, (here vamshikrishna-desktop
> is
> > the master and namenode machine)
> >
> >
> >
> > 2011-11-29 11:24:36,714 ERROR org.apache.hadoop.hbase.HServerAddress:
> Could
> > not resolve the DNS name of vamshikrishna-desktop
> > 2011-11-29 11:24:36,716 FATAL
> > org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region
> server
> > serverName=vamshi-laptop,60020,1322546076276, load=(requests=0,
> regions=0,
> > usedHeap=0, maxHeap=0): Unhandled exception: hostname can't be null
> > java.lang.IllegalArgumentException: hostname can't be null
> > at java.net.InetSocketAddress.<init>(InetSocketAddress.java:121)
> > at
> >
> org.apache.hadoop.hbase.HServerAddress.getResolvedAddress(HServerAddress.java:108)
> > at
> org.apache.hadoop.hbase.HServerAddress.<init>(HServerAddress.java:64)
> > at
> >
> org.apache.hadoop.hbase.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:63)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMasterAddress(HRegionServer.java:1484)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1457)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
> > at java.lang.Thread.run(Thread.java:662)
> > 2011-11-29 11:24:36,719 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unhandled
> > exception: hostname can't be null
> > 2011-11-29 11:24:36,719 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> > server on 60020
> > 2011-11-29 11:24:36,749 INFO
> > org.apache.hadoop.hbase.regionserver.StoreFile: Allocating LruBlockCache
> > with maximum size 199.4m
> > 2011-11-29 11:24:36,775 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at:
> > vamshi-laptop,60020,1322546076276
> > 2011-11-29 11:24:36,775 DEBUG
> > org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker
> > org.apache.hadoop.hbase.catalog.CatalogTracker@da2cef
> > 2011-11-29 11:24:36,776 INFO org.apache.hadoop.hbase.regionserver.Leases:
> > regionserver60020 closing leases
> > 2011-11-29 11:24:36,776 INFO org.apache.hadoop.hbase.regionserver.Leases:
> > regionserver60020 closed leases
> > 2011-11-29 11:24:36,776 INFO
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> > Closed zookeeper sessionid=0x133ede31f3c0004
> > 2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ClientCnxn: EventThread
> > shut down
> > 2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ZooKeeper: Session:
> > 0x133ede31f3c0004 closed
> > 2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ClientCnxn: EventThread
> > shut down
> > 2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ZooKeeper: Session:
> > 0x133ede31f3c0005 closed
> > 2011-11-29 11:24:36,792 INFO
> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
> starting;
> > hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-14,5,main]
> > 2011-11-29 11:24:36,792 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
> hook
> > 2011-11-29 11:24:36,792 INFO
> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs shutdown
> > hook thread.
> > 2011-11-29 11:24:36,792 INFO
> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
> finished.
> >
> > what is the cause for problem here, please help.
> >
> > On Mon, Nov 28, 2011 at 7:28 PM, Lars George <la...@gmail.com>
> wrote:
> >
> >> Hi,
> >>
> >> Did you add the list of servers to the regionservers file in the
> >> $HBASE_HOME/conf/ dir? Are you using Cygwin? Or what else is your
> >> environment?
> >>
> >> Lars
> >>
> >> On Nov 26, 2011, at 7:37 AM, Vamshi Krishna wrote:
> >>
> >> > Hi i am running hbase on 3 machines, on one node master and
> regionserver,
> >> > on other two nodes regionservers. i ran bin/start-hbase.sh on master,
> all
> >> > the hbase daemons running on master node(HMaster, HQuorumpeer,
> >> > HRegionServer), But on the other regionserver machines i could not
> find
> >> any
> >> > hbase daemons running. According to my knowledge the HRegionserver
> >> daemons
> >> > should run on other machines automatically when i run
> bin/start-hbase.sh
> >> on
> >> > master node, right? But its not happening so.
> >> > More over its printing errors on my master node when i run
> >> > bin/start-hbase.sh
> >> >
> >> > hbase-regionserver1: bash: line 0: cd: {HBASE_HOME}/bin/..: No such
> file
> >> or
> >> > directory
> >> > hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No such
> file
> >> > or directory
> >> > hbase-regionserver2: bash: line 0: cd: {HBASE_HOME}/bin/..: No such
> file
> >> or
> >> > directory
> >> > hbase-regionserver2: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No such
> file
> >> > or directory ,
> >> >
> >> > but i could find out {HBASE_HOME}/bin/hbase-daemon.sh on both
> machines.
> >> > Infact the path of the {HBASE_HOME} folder on each of the respective
> >> > machines machines is same. i.e
> >> > /home/hduser/Documents/HBASE_SOFTWRAE/hbase-0.90.4
> >> >
> >> > please can anybody help?
> >> >
> >> >
> >> > --
> >> > *Regards*
> >> > *
> >> > Vamshi Krishna
> >> > *
> >>
> >>
> >
> >
> > --
> > *Regards*
> > *
> > Vamshi Krishna
> > *
> >
>
--
*Regards*
*
Vamshi Krishna
*
Re: hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No
such file or directory
Posted by Jean-Daniel Cryans <jd...@apache.org>.
You posted this in two threads, please refrain from doing this in the future.
>From what I can read in there, it tried to get the master address
that's supposed to be in zookeeper but it failed because it was
missing and then died. The way it's handled is a bit ugly but the
effect is the same, take a look at the master log to see what's going
on.
J-D
On Mon, Nov 28, 2011 at 10:33 PM, Vamshi Krishna <va...@gmail.com> wrote:
> Hi Lars,
> i am not using cygwin, i am using 3 ubuntu-10.04 machines.
> Finally that problem i mentioned got resolved i.e now i can see the
> following after i run bin/start-hbase.sh on my master machine,
>
> hbase-master: starting zookeeper, logging to
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-zookeeper-vamshikrishna-desktop.out
> starting master, logging to
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/logs/hbase-hduser-master-vamshikrishna-desktop.out
> hbase-regionserver2: starting regionserver, logging to
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.out
> hbase-regionserver1: starting regionserver, logging to
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.out
>
> hbase-master: starting regionserver, logging to
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshikrishna-desktop.out
>
>
>
> But still , if i do jps on both region server machines, HRegionServer
> daemon is not seen.And i checked the logs on both the regionservere
> machines, i.e
>
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.log
> ---(hbase-regionserver1)
> and
> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.log
> ---(hbase-regionserver2)
>
> i found the same in both .it is like this, (here vamshikrishna-desktop is
> the master and namenode machine)
>
>
>
> 2011-11-29 11:24:36,714 ERROR org.apache.hadoop.hbase.HServerAddress: Could
> not resolve the DNS name of vamshikrishna-desktop
> 2011-11-29 11:24:36,716 FATAL
> org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
> serverName=vamshi-laptop,60020,1322546076276, load=(requests=0, regions=0,
> usedHeap=0, maxHeap=0): Unhandled exception: hostname can't be null
> java.lang.IllegalArgumentException: hostname can't be null
> at java.net.InetSocketAddress.<init>(InetSocketAddress.java:121)
> at
> org.apache.hadoop.hbase.HServerAddress.getResolvedAddress(HServerAddress.java:108)
> at org.apache.hadoop.hbase.HServerAddress.<init>(HServerAddress.java:64)
> at
> org.apache.hadoop.hbase.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:63)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMasterAddress(HRegionServer.java:1484)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1457)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
> at java.lang.Thread.run(Thread.java:662)
> 2011-11-29 11:24:36,719 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unhandled
> exception: hostname can't be null
> 2011-11-29 11:24:36,719 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> server on 60020
> 2011-11-29 11:24:36,749 INFO
> org.apache.hadoop.hbase.regionserver.StoreFile: Allocating LruBlockCache
> with maximum size 199.4m
> 2011-11-29 11:24:36,775 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at:
> vamshi-laptop,60020,1322546076276
> 2011-11-29 11:24:36,775 DEBUG
> org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker
> org.apache.hadoop.hbase.catalog.CatalogTracker@da2cef
> 2011-11-29 11:24:36,776 INFO org.apache.hadoop.hbase.regionserver.Leases:
> regionserver60020 closing leases
> 2011-11-29 11:24:36,776 INFO org.apache.hadoop.hbase.regionserver.Leases:
> regionserver60020 closed leases
> 2011-11-29 11:24:36,776 INFO
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Closed zookeeper sessionid=0x133ede31f3c0004
> 2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ClientCnxn: EventThread
> shut down
> 2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ZooKeeper: Session:
> 0x133ede31f3c0004 closed
> 2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ClientCnxn: EventThread
> shut down
> 2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ZooKeeper: Session:
> 0x133ede31f3c0005 closed
> 2011-11-29 11:24:36,792 INFO
> org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting;
> hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-14,5,main]
> 2011-11-29 11:24:36,792 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown hook
> 2011-11-29 11:24:36,792 INFO
> org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs shutdown
> hook thread.
> 2011-11-29 11:24:36,792 INFO
> org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook finished.
>
> what is the cause for problem here, please help.
>
> On Mon, Nov 28, 2011 at 7:28 PM, Lars George <la...@gmail.com> wrote:
>
>> Hi,
>>
>> Did you add the list of servers to the regionservers file in the
>> $HBASE_HOME/conf/ dir? Are you using Cygwin? Or what else is your
>> environment?
>>
>> Lars
>>
>> On Nov 26, 2011, at 7:37 AM, Vamshi Krishna wrote:
>>
>> > Hi i am running hbase on 3 machines, on one node master and regionserver,
>> > on other two nodes regionservers. i ran bin/start-hbase.sh on master, all
>> > the hbase daemons running on master node(HMaster, HQuorumpeer,
>> > HRegionServer), But on the other regionserver machines i could not find
>> any
>> > hbase daemons running. According to my knowledge the HRegionserver
>> daemons
>> > should run on other machines automatically when i run bin/start-hbase.sh
>> on
>> > master node, right? But its not happening so.
>> > More over its printing errors on my master node when i run
>> > bin/start-hbase.sh
>> >
>> > hbase-regionserver1: bash: line 0: cd: {HBASE_HOME}/bin/..: No such file
>> or
>> > directory
>> > hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No such file
>> > or directory
>> > hbase-regionserver2: bash: line 0: cd: {HBASE_HOME}/bin/..: No such file
>> or
>> > directory
>> > hbase-regionserver2: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No such file
>> > or directory ,
>> >
>> > but i could find out {HBASE_HOME}/bin/hbase-daemon.sh on both machines.
>> > Infact the path of the {HBASE_HOME} folder on each of the respective
>> > machines machines is same. i.e
>> > /home/hduser/Documents/HBASE_SOFTWRAE/hbase-0.90.4
>> >
>> > please can anybody help?
>> >
>> >
>> > --
>> > *Regards*
>> > *
>> > Vamshi Krishna
>> > *
>>
>>
>
>
> --
> *Regards*
> *
> Vamshi Krishna
> *
>
Re: hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No
such file or directory
Posted by Vamshi Krishna <va...@gmail.com>.
Hi Lars,
i am not using cygwin, i am using 3 ubuntu-10.04 machines.
Finally that problem i mentioned got resolved i.e now i can see the
following after i run bin/start-hbase.sh on my master machine,
hbase-master: starting zookeeper, logging to
/home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-zookeeper-vamshikrishna-desktop.out
starting master, logging to
/home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/logs/hbase-hduser-master-vamshikrishna-desktop.out
hbase-regionserver2: starting regionserver, logging to
/home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.out
hbase-regionserver1: starting regionserver, logging to
/home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.out
hbase-master: starting regionserver, logging to
/home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshikrishna-desktop.out
But still , if i do jps on both region server machines, HRegionServer
daemon is not seen.And i checked the logs on both the regionservere
machines, i.e
/home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.log
---(hbase-regionserver1)
and
/home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.log
---(hbase-regionserver2)
i found the same in both .it is like this, (here vamshikrishna-desktop is
the master and namenode machine)
2011-11-29 11:24:36,714 ERROR org.apache.hadoop.hbase.HServerAddress: Could
not resolve the DNS name of vamshikrishna-desktop
2011-11-29 11:24:36,716 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
serverName=vamshi-laptop,60020,1322546076276, load=(requests=0, regions=0,
usedHeap=0, maxHeap=0): Unhandled exception: hostname can't be null
java.lang.IllegalArgumentException: hostname can't be null
at java.net.InetSocketAddress.<init>(InetSocketAddress.java:121)
at
org.apache.hadoop.hbase.HServerAddress.getResolvedAddress(HServerAddress.java:108)
at org.apache.hadoop.hbase.HServerAddress.<init>(HServerAddress.java:64)
at
org.apache.hadoop.hbase.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:63)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getMasterAddress(HRegionServer.java:1484)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1457)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
at java.lang.Thread.run(Thread.java:662)
2011-11-29 11:24:36,719 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unhandled
exception: hostname can't be null
2011-11-29 11:24:36,719 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60020
2011-11-29 11:24:36,749 INFO
org.apache.hadoop.hbase.regionserver.StoreFile: Allocating LruBlockCache
with maximum size 199.4m
2011-11-29 11:24:36,775 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at:
vamshi-laptop,60020,1322546076276
2011-11-29 11:24:36,775 DEBUG
org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker
org.apache.hadoop.hbase.catalog.CatalogTracker@da2cef
2011-11-29 11:24:36,776 INFO org.apache.hadoop.hbase.regionserver.Leases:
regionserver60020 closing leases
2011-11-29 11:24:36,776 INFO org.apache.hadoop.hbase.regionserver.Leases:
regionserver60020 closed leases
2011-11-29 11:24:36,776 INFO
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Closed zookeeper sessionid=0x133ede31f3c0004
2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ZooKeeper: Session:
0x133ede31f3c0004 closed
2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ZooKeeper: Session:
0x133ede31f3c0005 closed
2011-11-29 11:24:36,792 INFO
org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting;
hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-14,5,main]
2011-11-29 11:24:36,792 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown hook
2011-11-29 11:24:36,792 INFO
org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs shutdown
hook thread.
2011-11-29 11:24:36,792 INFO
org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook finished.
what is the cause for problem here, please help.
On Mon, Nov 28, 2011 at 7:28 PM, Lars George <la...@gmail.com> wrote:
> Hi,
>
> Did you add the list of servers to the regionservers file in the
> $HBASE_HOME/conf/ dir? Are you using Cygwin? Or what else is your
> environment?
>
> Lars
>
> On Nov 26, 2011, at 7:37 AM, Vamshi Krishna wrote:
>
> > Hi i am running hbase on 3 machines, on one node master and regionserver,
> > on other two nodes regionservers. i ran bin/start-hbase.sh on master, all
> > the hbase daemons running on master node(HMaster, HQuorumpeer,
> > HRegionServer), But on the other regionserver machines i could not find
> any
> > hbase daemons running. According to my knowledge the HRegionserver
> daemons
> > should run on other machines automatically when i run bin/start-hbase.sh
> on
> > master node, right? But its not happening so.
> > More over its printing errors on my master node when i run
> > bin/start-hbase.sh
> >
> > hbase-regionserver1: bash: line 0: cd: {HBASE_HOME}/bin/..: No such file
> or
> > directory
> > hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No such file
> > or directory
> > hbase-regionserver2: bash: line 0: cd: {HBASE_HOME}/bin/..: No such file
> or
> > directory
> > hbase-regionserver2: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No such file
> > or directory ,
> >
> > but i could find out {HBASE_HOME}/bin/hbase-daemon.sh on both machines.
> > Infact the path of the {HBASE_HOME} folder on each of the respective
> > machines machines is same. i.e
> > /home/hduser/Documents/HBASE_SOFTWRAE/hbase-0.90.4
> >
> > please can anybody help?
> >
> >
> > --
> > *Regards*
> > *
> > Vamshi Krishna
> > *
>
>
--
*Regards*
*
Vamshi Krishna
*
Re: hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No such file or directory
Posted by Lars George <la...@gmail.com>.
Hi,
Did you add the list of servers to the regionservers file in the $HBASE_HOME/conf/ dir? Are you using Cygwin? Or what else is your environment?
Lars
On Nov 26, 2011, at 7:37 AM, Vamshi Krishna wrote:
> Hi i am running hbase on 3 machines, on one node master and regionserver,
> on other two nodes regionservers. i ran bin/start-hbase.sh on master, all
> the hbase daemons running on master node(HMaster, HQuorumpeer,
> HRegionServer), But on the other regionserver machines i could not find any
> hbase daemons running. According to my knowledge the HRegionserver daemons
> should run on other machines automatically when i run bin/start-hbase.sh on
> master node, right? But its not happening so.
> More over its printing errors on my master node when i run
> bin/start-hbase.sh
>
> hbase-regionserver1: bash: line 0: cd: {HBASE_HOME}/bin/..: No such file or
> directory
> hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No such file
> or directory
> hbase-regionserver2: bash: line 0: cd: {HBASE_HOME}/bin/..: No such file or
> directory
> hbase-regionserver2: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No such file
> or directory ,
>
> but i could find out {HBASE_HOME}/bin/hbase-daemon.sh on both machines.
> Infact the path of the {HBASE_HOME} folder on each of the respective
> machines machines is same. i.e
> /home/hduser/Documents/HBASE_SOFTWRAE/hbase-0.90.4
>
> please can anybody help?
>
>
> --
> *Regards*
> *
> Vamshi Krishna
> *