You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jean-Daniel Cryans <jd...@apache.org> on 2010/03/11 19:19:09 UTC
Re: region server appearing twice on HBase Master page
Bringing the discussion in hbase-user
That usually happens after a DNS hiccup. There's a fix for that in
https://issues.apache.org/jira/browse/HBASE-2174
J-D
On Wed, Mar 10, 2010 at 1:41 PM, Ted Yu <yu...@gmail.com> wrote:
> I noticed two lines for the same region server on HBase Master page:
> X.com:60030 1268160765854 requests=0, regions=16, usedHeap=1068,
> maxHeap=6127
> X.com:60030 1268250726442 requests=21, regions=9, usedHeap=1258,
> maxHeap=6127
>
> I checked there is only one
> org.apache.hadoop.hbase.regionserver.HRegionServer instance running on that
> machine.
>
> This is from region server log:
>
> 2010-03-10 13:25:38,157 ERROR [IPC Server handler 43 on 60020]
> regionserver.HRegionServer(844):
> org.apache.hadoop.hbase.NotServingRegionException: ruletable,,1268083966723
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2307)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1784)
> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> 2010-03-10 13:25:38,189 ERROR [IPC Server handler 0 on 60020]
> regionserver.HRegionServer(844):
> org.apache.hadoop.hbase.NotServingRegionException: ruletable,,1268083966723
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2307)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1784)
> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>
> If you know how to troubleshoot, please share.
>
RE: region server appearing twice on HBase Master page
Posted by Michael Segel <mi...@hotmail.com>.
> Date: Thu, 11 Mar 2010 13:55:20 -0800
> Subject: Re: region server appearing twice on HBase Master page
> From: stack@duboce.net
> To: hbase-user@hadoop.apache.org
>
> On Thu, Mar 11, 2010 at 1:48 PM, Michael Segel
> <mi...@hotmail.com> wrote:
> >
> > Hey!
> > The patch appears to be working, but can anyone give any more information on what could be causing the DNS 'hiccup'?
> >
>
> Isn't this a question for your ops team? Why a lookup gives different
> answers at different times (IIUC)?
>
>
Well does the expression YMMV mean anything when it comes to interpreting their response? ;-)
Which is why I had to ask. Depending on how the question is asked, and who you ask, you may get a different response.
They're monitoring the system and they see that HBase is sending ipv6 requests since its up but not in use. The thought was that it could have been the problem. Not sure if the 'hiccup' you mentioned has anything to do with this.
> > When we shut off IPV6 and rebooted, HBase wouldn't come up, we reversed the changes and could come up.
> >
>
> We don't explicity depend on ipv6. What was the issue that you were seeing?
>
> St.Ack
This is the weird thing...
On the name node, it would try to connect to itself and it would fail. (connection refused.) Don't ask me why, but somehow it was trying to start a region server on the master. (And no, the machine isn't listed as a region server)
Then when I tried to shut it down and even restart it, I would see that it did start zookeeper and the region servers were up. Just no communication between the master and the nodes.
They backed the change out and again, I can shut down the region servers using stop-hbase.sh, which does stop the zookeepers but the region servers now stay up.
Not sure what is happening, of course I don't know if ops is playing around with the configuration.
I realize that this is less than helpful, but it should explain some of the frustration and difficulty when trying to debug a less than optimal situation.
-Mike
_________________________________________________________________
Hotmail: Free, trusted and rich email service.
http://clk.atdmt.com/GBL/go/201469228/direct/01/
Re: region server appearing twice on HBase Master page
Posted by Stack <st...@duboce.net>.
On Thu, Mar 11, 2010 at 1:48 PM, Michael Segel
<mi...@hotmail.com> wrote:
>
> Hey!
> The patch appears to be working, but can anyone give any more information on what could be causing the DNS 'hiccup'?
>
Isn't this a question for your ops team? Why a lookup gives different
answers at different times (IIUC)?
> When we shut off IPV6 and rebooted, HBase wouldn't come up, we reversed the changes and could come up.
>
We don't explicity depend on ipv6. What was the issue that you were seeing?
St.Ack
RE: region server appearing twice on HBase Master page
Posted by Michael Segel <mi...@hotmail.com>.
> Date: Thu, 11 Mar 2010 12:08:34 -0800
> Subject: Re: region server appearing twice on HBase Master page
> From: stack@duboce.net
> To: hbase-user@hadoop.apache.org
>
> I just applied hbase-2174 to branch and trunk.
> St.Ack
>
> On Thu, Mar 11, 2010 at 10:19 AM, Jean-Daniel Cryans
> <jd...@apache.org> wrote:
> > Bringing the discussion in hbase-user
> >
> > That usually happens after a DNS hiccup. There's a fix for that in
> > https://issues.apache.org/jira/browse/HBASE-2174
> >
> > J-D
Hey!
The patch appears to be working, but can anyone give any more information on what could be causing the DNS 'hiccup'?
I mean is it that there is an inconsistency between two different DNS servers?
Is it a difference between /etc/hosts and the DNS service?
Does IPV6 have any impact on this? (Seems we have IPV6 turned on but its not being used.)
When we shut off IPV6 and rebooted, HBase wouldn't come up, we reversed the changes and could come up.
Thx
-Mike
_________________________________________________________________
Hotmail: Trusted email with Microsoft’s powerful SPAM protection.
http://clk.atdmt.com/GBL/go/201469226/direct/01/
Re: region server appearing twice on HBase Master page
Posted by Stack <st...@duboce.net>.
I just applied hbase-2174 to branch and trunk.
St.Ack
On Thu, Mar 11, 2010 at 10:19 AM, Jean-Daniel Cryans
<jd...@apache.org> wrote:
> Bringing the discussion in hbase-user
>
> That usually happens after a DNS hiccup. There's a fix for that in
> https://issues.apache.org/jira/browse/HBASE-2174
>
> J-D
>
> On Wed, Mar 10, 2010 at 1:41 PM, Ted Yu <yu...@gmail.com> wrote:
>> I noticed two lines for the same region server on HBase Master page:
>> X.com:60030 1268160765854 requests=0, regions=16, usedHeap=1068,
>> maxHeap=6127
>> X.com:60030 1268250726442 requests=21, regions=9, usedHeap=1258,
>> maxHeap=6127
>>
>> I checked there is only one
>> org.apache.hadoop.hbase.regionserver.HRegionServer instance running on that
>> machine.
>>
>> This is from region server log:
>>
>> 2010-03-10 13:25:38,157 ERROR [IPC Server handler 43 on 60020]
>> regionserver.HRegionServer(844):
>> org.apache.hadoop.hbase.NotServingRegionException: ruletable,,1268083966723
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2307)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1784)
>> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>> 2010-03-10 13:25:38,189 ERROR [IPC Server handler 0 on 60020]
>> regionserver.HRegionServer(844):
>> org.apache.hadoop.hbase.NotServingRegionException: ruletable,,1268083966723
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2307)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1784)
>> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>>
>> If you know how to troubleshoot, please share.
>>
>
Re: region server appearing twice on HBase Master page
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Yes, servers and clients will all need to be on 0.20.4... that's the
tradeoff for more flexibility in the future (see
http://issues.apache.org/jira/browse/HBASE-2219).
J-D
On Thu, Mar 11, 2010 at 10:52 AM, Ted Yu <yu...@gmail.com> wrote:
> That makes sense.
> This means we have to replace hbase client when we upgrade to 0.20.4, right
> ?
>
> On Thu, Mar 11, 2010 at 10:36 AM, Jean-Daniel Cryans <jd...@apache.org>
> wrote:
>>
>> We are about to commit it to 0.20.4 since we voted on it breaking RPC
>> compatibility.
>>
>> In this case the DNS lookup gives a different address, so the master
>> treats it as a different region server and gives it a new startcode.
>> In the web ui the two lines should have the same address, but not the
>> same code at the end.
>>
>> J-D
>>
>> On Thu, Mar 11, 2010 at 10:33 AM, Ted Yu <yu...@gmail.com> wrote:
>> > 0.20.5 seems a bit far in the future :-)
>> >
>> > What I couldn't explain is that why serversToServerInfo, backed by
>> > ConcurrentHashMap, would contain two entries with same key - X.com.
>> >
>> > On Thu, Mar 11, 2010 at 10:19 AM, Jean-Daniel Cryans
>> > <jd...@apache.org>wrote:
>> >
>> >> Bringing the discussion in hbase-user
>> >>
>> >> That usually happens after a DNS hiccup. There's a fix for that in
>> >> https://issues.apache.org/jira/browse/HBASE-2174
>> >>
>> >> J-D
>> >>
>> >> On Wed, Mar 10, 2010 at 1:41 PM, Ted Yu <yu...@gmail.com> wrote:
>> >> > I noticed two lines for the same region server on HBase Master page:
>> >> > X.com:60030 1268160765854 requests=0, regions=16,
>> >> > usedHeap=1068,
>> >> > maxHeap=6127
>> >> > X.com:60030 1268250726442 requests=21, regions=9,
>> >> > usedHeap=1258,
>> >> > maxHeap=6127
>> >> >
>> >> > I checked there is only one
>> >> > org.apache.hadoop.hbase.regionserver.HRegionServer instance running
>> >> > on
>> >> that
>> >> > machine.
>> >> >
>> >> > This is from region server log:
>> >> >
>> >> > 2010-03-10 13:25:38,157 ERROR [IPC Server handler 43 on 60020]
>> >> > regionserver.HRegionServer(844):
>> >> > org.apache.hadoop.hbase.NotServingRegionException:
>> >> ruletable,,1268083966723
>> >> > at
>> >> >
>> >>
>> >> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2307)
>> >> > at
>> >> >
>> >>
>> >> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1784)
>> >> > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown
>> >> > Source)
>> >> > at
>> >> >
>> >>
>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >> > at
>> >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
>> >> > at
>> >> >
>> >> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>> >> > 2010-03-10 13:25:38,189 ERROR [IPC Server handler 0 on 60020]
>> >> > regionserver.HRegionServer(844):
>> >> > org.apache.hadoop.hbase.NotServingRegionException:
>> >> ruletable,,1268083966723
>> >> > at
>> >> >
>> >>
>> >> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2307)
>> >> > at
>> >> >
>> >>
>> >> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1784)
>> >> > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown
>> >> > Source)
>> >> > at
>> >> >
>> >>
>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >> > at
>> >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
>> >> > at
>> >> >
>> >> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>> >> >
>> >> > If you know how to troubleshoot, please share.
>> >> >
>> >>
>> >
>
>
Re: region server appearing twice on HBase Master page
Posted by Jean-Daniel Cryans <jd...@apache.org>.
We are about to commit it to 0.20.4 since we voted on it breaking RPC
compatibility.
In this case the DNS lookup gives a different address, so the master
treats it as a different region server and gives it a new startcode.
In the web ui the two lines should have the same address, but not the
same code at the end.
J-D
On Thu, Mar 11, 2010 at 10:33 AM, Ted Yu <yu...@gmail.com> wrote:
> 0.20.5 seems a bit far in the future :-)
>
> What I couldn't explain is that why serversToServerInfo, backed by
> ConcurrentHashMap, would contain two entries with same key - X.com.
>
> On Thu, Mar 11, 2010 at 10:19 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> Bringing the discussion in hbase-user
>>
>> That usually happens after a DNS hiccup. There's a fix for that in
>> https://issues.apache.org/jira/browse/HBASE-2174
>>
>> J-D
>>
>> On Wed, Mar 10, 2010 at 1:41 PM, Ted Yu <yu...@gmail.com> wrote:
>> > I noticed two lines for the same region server on HBase Master page:
>> > X.com:60030 1268160765854 requests=0, regions=16, usedHeap=1068,
>> > maxHeap=6127
>> > X.com:60030 1268250726442 requests=21, regions=9, usedHeap=1258,
>> > maxHeap=6127
>> >
>> > I checked there is only one
>> > org.apache.hadoop.hbase.regionserver.HRegionServer instance running on
>> that
>> > machine.
>> >
>> > This is from region server log:
>> >
>> > 2010-03-10 13:25:38,157 ERROR [IPC Server handler 43 on 60020]
>> > regionserver.HRegionServer(844):
>> > org.apache.hadoop.hbase.NotServingRegionException:
>> ruletable,,1268083966723
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2307)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1784)
>> > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
>> > at
>> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>> > 2010-03-10 13:25:38,189 ERROR [IPC Server handler 0 on 60020]
>> > regionserver.HRegionServer(844):
>> > org.apache.hadoop.hbase.NotServingRegionException:
>> ruletable,,1268083966723
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2307)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1784)
>> > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
>> > at
>> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>> >
>> > If you know how to troubleshoot, please share.
>> >
>>
>
Re: region server appearing twice on HBase Master page
Posted by Ted Yu <yu...@gmail.com>.
0.20.5 seems a bit far in the future :-)
What I couldn't explain is that why serversToServerInfo, backed by
ConcurrentHashMap, would contain two entries with same key - X.com.
On Thu, Mar 11, 2010 at 10:19 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> Bringing the discussion in hbase-user
>
> That usually happens after a DNS hiccup. There's a fix for that in
> https://issues.apache.org/jira/browse/HBASE-2174
>
> J-D
>
> On Wed, Mar 10, 2010 at 1:41 PM, Ted Yu <yu...@gmail.com> wrote:
> > I noticed two lines for the same region server on HBase Master page:
> > X.com:60030 1268160765854 requests=0, regions=16, usedHeap=1068,
> > maxHeap=6127
> > X.com:60030 1268250726442 requests=21, regions=9, usedHeap=1258,
> > maxHeap=6127
> >
> > I checked there is only one
> > org.apache.hadoop.hbase.regionserver.HRegionServer instance running on
> that
> > machine.
> >
> > This is from region server log:
> >
> > 2010-03-10 13:25:38,157 ERROR [IPC Server handler 43 on 60020]
> > regionserver.HRegionServer(844):
> > org.apache.hadoop.hbase.NotServingRegionException:
> ruletable,,1268083966723
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2307)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1784)
> > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> > 2010-03-10 13:25:38,189 ERROR [IPC Server handler 0 on 60020]
> > regionserver.HRegionServer(844):
> > org.apache.hadoop.hbase.NotServingRegionException:
> ruletable,,1268083966723
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2307)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1784)
> > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> >
> > If you know how to troubleshoot, please share.
> >
>