You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Sasha Dolgy <sa...@gmail.com> on 2009/05/15 10:53:37 UTC
HRegionServer: Failed openScanner
Hi there,
Thought I would run a count 'table-name' to see how many records are in a
table. The count didn't work so I went and took a look in the regionserver
log file and see the below. Now, to be honest, not quite sure why it's just
stopped working. The lines are from the start of the log file (from may
15). The log file from May 14 has no errors in it. Is there any way I can
enable debugging to find out why it's stopped working? I had added this in
the past 36 hours but it didn't appear to cause any problems and was working
fine for atleast 24 hours:
<property>
<name>hbase.regionserver.class</name>
<value>org.apache.hadoop.hbase.ipc.IndexedRegionInterface</value>
<description>enable indexing</description>
</property>
<property>
<name>hbase.regionserver.impl</name>
<value>org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegionServer</value>
<description>enable indexing</description>
</property>
$HBASE_HOME/bin/stop-all.sh and $HBASE_HOME/bin/start-all.sh fixed
everything and it's all working fine again.
2009-05-15 00:55:53,090 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
master for 189261 milliseconds - retrying
2009-05-15 00:55:56,789 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_CALL_SERVER_STARTUP:
safeMode=false
2009-05-15 00:55:57,249 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
2009-05-15 00:55:57,252 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60020, call openScanner([B@233dfbc1, [[B@3a5b4dfa, [B@405c7604,
9223372036854775807, null) from 127.0.0.1:47277: error:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
2009-05-15 00:56:57,751 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
--
Sasha Dolgy
sasha.dolgy@gmail.com
Re: HRegionServer: Failed openScanner
Posted by Andrew Purtell <ap...@apache.org>.
If they are both on the same server then almost surely the issue is swapping or CPU overcommitment. That said, additional information from your logs, if germane, may shed more light on this incident.
Best regards,
- Andy
________________________________
From: Sasha Dolgy <sd...@gmail.com>
To: hbase-user@hadoop.apache.org
Sent: Friday, May 15, 2009 11:32:56 AM
Subject: Re: HRegionServer: Failed openScanner
Ok, i'll go take a look. They are both on the local server so network
issues shouldn't be a cause. Cheers though, i'll go look at the JIRA link.
If I find anything else i'll post here.
thanks
-sd
On Fri, May 15, 2009 at 6:18 PM, Andrew Purtell <ap...@apache.org> wrote:
> The region server hosting META could not communicate with the master for a
> very long time. Some kind of network issue? Any entries in the region server
> logs above this one
>
> > 2009-05-15 00:55:53,090 WARN
> > org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
> > master for 189261 milliseconds - retrying
>
> which may be relevant? Anything about sleeping too long?
>
> Related, there were some bugs that I am aware of preventing recovery if
> META in particular goes away but they should be fixed for 0.20 as of
> https://issues.apache.org/jira/browse/HBASE-1362 .
>
> - Andy
>
Re: HRegionServer: Failed openScanner
Posted by zsongbo <zs...@gmail.com>.
Please refer another mail thread: "RegionServer failure and recovery take a
long time". J-D has given me many good advices.
1. set low value to "swappiness" to avoid too many swap. for example 20 or
10.
2. use some jvm GC options, I am using -XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode,
and would and -XX:ParallelGCThreads=8 later.
2. use a patch from J-D: https://issues.apache.org/jira/browse/HBASE-1008
Schubert
On Sat, May 16, 2009 at 5:16 AM, stack <st...@duboce.net> wrote:
> Andy's base point is that you've probably overloaded your setup. What are
> you hoping to achieve with this setup of one machine?
>
> You've followed the 'Getting Started' section in hbase documentation? Has
> some configuration you need. Enable the troubleshooting suggested
> configurations too if you want to remove lack of resources or incorrect
> timeouts as cause. You should enable DEBUG too. Will make your logs
> richer
> in detail and will help with the diagnosis.
>
> Thanks,
> St.Ack
>
>
> On Fri, May 15, 2009 at 11:58 AM, Sasha Dolgy <sd...@gmail.com> wrote:
>
> > Hi Andy,
> > I've sent you an email with a link to a tar file with the logs. To be
> > honest, for the most part this is default out of the box. To this point
> > this is the first problem with over 150k writes to HBase. After i
> stopped
> > /
> > started HBase again everything is going fine...
> >
> > I haven't looked at the troubleshooting page yet, because well, i'm not
> > quite sure what to trouble shoot. Finding it hard to identify an actual
> > problem....other then seeing stack traces and it not working.
> >
> > -sd
> >
> > On Fri, May 15, 2009 at 7:54 PM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> > > This is almost surely resource overcommitment as cause: CPU and/or
> > memory,
> > > leading to thread starvation. We observe the JVM scheduler is unfair at
> > high
> > > load, and swap, especially if JVM heap is paged out when a GC cycle
> > happens,
> > > can also be similarly deadly. Give other details in this thread, I
> > suspect
> > > swap. What JVM options are you running with? Have you looked at the GC
> > > related tips on the troubleshooting page up on the wiki?
> > > http://wiki.apache.org/hadoop/Hbase/Troubleshooting
> > >
> > > Best regards,
> > >
> > > - Andy
> > >
> > >
> > >
> > >
> > > ________________________________
> > > From: Sasha Dolgy <sd...@gmail.com>
> > > To: hbase-user@hadoop.apache.org
> > > Sent: Friday, May 15, 2009 11:38:01 AM
> > > Subject: Re: HRegionServer: Failed openScanner
> > >
> > > In the region server logs I see messages from the 14th:
> > > 2009-05-14 22:47:28,840 INFO
> > org.apache.hadoop.hbase.regionserver.HRegion:
> > > starting compaction on region syslog,,1242260881586
> > > 2009-05-14 22:47:43,976 INFO
> > org.apache.hadoop.hbase.regionserver.HRegion:
> > > compaction completed on region syslog,,1242260881586 in 15sec
> > >
> > > then no log entries until the 15th when the error happens:
> > >
> > > 2009-05-15 00:55:51,568 WARN org.apache.hadoop.hbase.util.Sleeper: We
> > slept
> > > 189138ms, ten times longer than scheduled: 10000
> > > 2009-05-15 00:55:52,334 WARN org.apache.hadoop.hbase.util.Sleeper: We
> > slept
> > > 188348ms, ten times longer than scheduled: 3000
> > > 2009-05-15 00:55:53,090 WARN
> > > org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
> > > master for 189261 milliseconds - retrying
> > > 2009-05-15 00:55:56,789 INFO
> > > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > MSG_CALL_SERVER_STARTUP:
> > > safeMode=false
> > > 2009-05-15 00:55:57,249 ERROR
> > > org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
> > > org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > > at
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at
> > > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> > > at
> > >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
> > >
> > >
> > >
> > > On Fri, May 15, 2009 at 7:32 PM, Sasha Dolgy <sd...@gmail.com> wrote:
> > >
> > > > Ok, i'll go take a look. They are both on the local server so
> network
> > > > issues shouldn't be a cause. Cheers though, i'll go look at the JIRA
> > > link.
> > > > If I find anything else i'll post here.
> > > > thanks
> > > > -sd
> > > >
> > > > On Fri, May 15, 2009 at 6:18 PM, Andrew Purtell <apurtell@apache.org
> > > >wrote:
> > > >
> > > >> The region server hosting META could not communicate with the master
> > for
> > > a
> > > >> very long time. Some kind of network issue? Any entries in the
> region
> > > server
> > > >> logs above this one
> > > >>
> > > >> > 2009-05-15 00:55:53,090 WARN
> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: unable to
> report
> > > to
> > > >> > master for 189261 milliseconds - retrying
> > > >>
> > > >> which may be relevant? Anything about sleeping too long?
> > > >>
> > > >> Related, there were some bugs that I am aware of preventing recovery
> > if
> > > >> META in particular goes away but they should be fixed for 0.20 as of
> > > >> https://issues.apache.org/jira/browse/HBASE-1362 .
> > > >>
> > > >> - Andy
> > > >>
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> > --
> > Sasha Dolgy
> > sasha.dolgy@gmail.com
> >
>
Re: HRegionServer: Failed openScanner
Posted by stack <st...@duboce.net>.
Andy's base point is that you've probably overloaded your setup. What are
you hoping to achieve with this setup of one machine?
You've followed the 'Getting Started' section in hbase documentation? Has
some configuration you need. Enable the troubleshooting suggested
configurations too if you want to remove lack of resources or incorrect
timeouts as cause. You should enable DEBUG too. Will make your logs richer
in detail and will help with the diagnosis.
Thanks,
St.Ack
On Fri, May 15, 2009 at 11:58 AM, Sasha Dolgy <sd...@gmail.com> wrote:
> Hi Andy,
> I've sent you an email with a link to a tar file with the logs. To be
> honest, for the most part this is default out of the box. To this point
> this is the first problem with over 150k writes to HBase. After i stopped
> /
> started HBase again everything is going fine...
>
> I haven't looked at the troubleshooting page yet, because well, i'm not
> quite sure what to trouble shoot. Finding it hard to identify an actual
> problem....other then seeing stack traces and it not working.
>
> -sd
>
> On Fri, May 15, 2009 at 7:54 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > This is almost surely resource overcommitment as cause: CPU and/or
> memory,
> > leading to thread starvation. We observe the JVM scheduler is unfair at
> high
> > load, and swap, especially if JVM heap is paged out when a GC cycle
> happens,
> > can also be similarly deadly. Give other details in this thread, I
> suspect
> > swap. What JVM options are you running with? Have you looked at the GC
> > related tips on the troubleshooting page up on the wiki?
> > http://wiki.apache.org/hadoop/Hbase/Troubleshooting
> >
> > Best regards,
> >
> > - Andy
> >
> >
> >
> >
> > ________________________________
> > From: Sasha Dolgy <sd...@gmail.com>
> > To: hbase-user@hadoop.apache.org
> > Sent: Friday, May 15, 2009 11:38:01 AM
> > Subject: Re: HRegionServer: Failed openScanner
> >
> > In the region server logs I see messages from the 14th:
> > 2009-05-14 22:47:28,840 INFO
> org.apache.hadoop.hbase.regionserver.HRegion:
> > starting compaction on region syslog,,1242260881586
> > 2009-05-14 22:47:43,976 INFO
> org.apache.hadoop.hbase.regionserver.HRegion:
> > compaction completed on region syslog,,1242260881586 in 15sec
> >
> > then no log entries until the 15th when the error happens:
> >
> > 2009-05-15 00:55:51,568 WARN org.apache.hadoop.hbase.util.Sleeper: We
> slept
> > 189138ms, ten times longer than scheduled: 10000
> > 2009-05-15 00:55:52,334 WARN org.apache.hadoop.hbase.util.Sleeper: We
> slept
> > 188348ms, ten times longer than scheduled: 3000
> > 2009-05-15 00:55:53,090 WARN
> > org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
> > master for 189261 milliseconds - retrying
> > 2009-05-15 00:55:56,789 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > MSG_CALL_SERVER_STARTUP:
> > safeMode=false
> > 2009-05-15 00:55:57,249 ERROR
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
> > org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
> >
> >
> >
> > On Fri, May 15, 2009 at 7:32 PM, Sasha Dolgy <sd...@gmail.com> wrote:
> >
> > > Ok, i'll go take a look. They are both on the local server so network
> > > issues shouldn't be a cause. Cheers though, i'll go look at the JIRA
> > link.
> > > If I find anything else i'll post here.
> > > thanks
> > > -sd
> > >
> > > On Fri, May 15, 2009 at 6:18 PM, Andrew Purtell <apurtell@apache.org
> > >wrote:
> > >
> > >> The region server hosting META could not communicate with the master
> for
> > a
> > >> very long time. Some kind of network issue? Any entries in the region
> > server
> > >> logs above this one
> > >>
> > >> > 2009-05-15 00:55:53,090 WARN
> > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report
> > to
> > >> > master for 189261 milliseconds - retrying
> > >>
> > >> which may be relevant? Anything about sleeping too long?
> > >>
> > >> Related, there were some bugs that I am aware of preventing recovery
> if
> > >> META in particular goes away but they should be fixed for 0.20 as of
> > >> https://issues.apache.org/jira/browse/HBASE-1362 .
> > >>
> > >> - Andy
> > >>
> > >
> >
> >
> >
> >
> >
>
>
>
> --
> Sasha Dolgy
> sasha.dolgy@gmail.com
>
Re: HRegionServer: Failed openScanner
Posted by Sasha Dolgy <sd...@gmail.com>.
Hi Andy,
I've sent you an email with a link to a tar file with the logs. To be
honest, for the most part this is default out of the box. To this point
this is the first problem with over 150k writes to HBase. After i stopped /
started HBase again everything is going fine...
I haven't looked at the troubleshooting page yet, because well, i'm not
quite sure what to trouble shoot. Finding it hard to identify an actual
problem....other then seeing stack traces and it not working.
-sd
On Fri, May 15, 2009 at 7:54 PM, Andrew Purtell <ap...@apache.org> wrote:
> This is almost surely resource overcommitment as cause: CPU and/or memory,
> leading to thread starvation. We observe the JVM scheduler is unfair at high
> load, and swap, especially if JVM heap is paged out when a GC cycle happens,
> can also be similarly deadly. Give other details in this thread, I suspect
> swap. What JVM options are you running with? Have you looked at the GC
> related tips on the troubleshooting page up on the wiki?
> http://wiki.apache.org/hadoop/Hbase/Troubleshooting
>
> Best regards,
>
> - Andy
>
>
>
>
> ________________________________
> From: Sasha Dolgy <sd...@gmail.com>
> To: hbase-user@hadoop.apache.org
> Sent: Friday, May 15, 2009 11:38:01 AM
> Subject: Re: HRegionServer: Failed openScanner
>
> In the region server logs I see messages from the 14th:
> 2009-05-14 22:47:28,840 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> starting compaction on region syslog,,1242260881586
> 2009-05-14 22:47:43,976 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> compaction completed on region syslog,,1242260881586 in 15sec
>
> then no log entries until the 15th when the error happens:
>
> 2009-05-15 00:55:51,568 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
> 189138ms, ten times longer than scheduled: 10000
> 2009-05-15 00:55:52,334 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
> 188348ms, ten times longer than scheduled: 3000
> 2009-05-15 00:55:53,090 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
> master for 189261 milliseconds - retrying
> 2009-05-15 00:55:56,789 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_CALL_SERVER_STARTUP:
> safeMode=false
> 2009-05-15 00:55:57,249 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
>
>
>
> On Fri, May 15, 2009 at 7:32 PM, Sasha Dolgy <sd...@gmail.com> wrote:
>
> > Ok, i'll go take a look. They are both on the local server so network
> > issues shouldn't be a cause. Cheers though, i'll go look at the JIRA
> link.
> > If I find anything else i'll post here.
> > thanks
> > -sd
> >
> > On Fri, May 15, 2009 at 6:18 PM, Andrew Purtell <apurtell@apache.org
> >wrote:
> >
> >> The region server hosting META could not communicate with the master for
> a
> >> very long time. Some kind of network issue? Any entries in the region
> server
> >> logs above this one
> >>
> >> > 2009-05-15 00:55:53,090 WARN
> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report
> to
> >> > master for 189261 milliseconds - retrying
> >>
> >> which may be relevant? Anything about sleeping too long?
> >>
> >> Related, there were some bugs that I am aware of preventing recovery if
> >> META in particular goes away but they should be fixed for 0.20 as of
> >> https://issues.apache.org/jira/browse/HBASE-1362 .
> >>
> >> - Andy
> >>
> >
>
>
>
>
>
--
Sasha Dolgy
sasha.dolgy@gmail.com
Re: HRegionServer: Failed openScanner
Posted by Andrew Purtell <ap...@apache.org>.
This is almost surely resource overcommitment as cause: CPU and/or memory, leading to thread starvation. We observe the JVM scheduler is unfair at high load, and swap, especially if JVM heap is paged out when a GC cycle happens, can also be similarly deadly. Give other details in this thread, I suspect swap. What JVM options are you running with? Have you looked at the GC related tips on the troubleshooting page up on the wiki? http://wiki.apache.org/hadoop/Hbase/Troubleshooting
Best regards,
- Andy
________________________________
From: Sasha Dolgy <sd...@gmail.com>
To: hbase-user@hadoop.apache.org
Sent: Friday, May 15, 2009 11:38:01 AM
Subject: Re: HRegionServer: Failed openScanner
In the region server logs I see messages from the 14th:
2009-05-14 22:47:28,840 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region syslog,,1242260881586
2009-05-14 22:47:43,976 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region syslog,,1242260881586 in 15sec
then no log entries until the 15th when the error happens:
2009-05-15 00:55:51,568 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
189138ms, ten times longer than scheduled: 10000
2009-05-15 00:55:52,334 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
188348ms, ten times longer than scheduled: 3000
2009-05-15 00:55:53,090 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
master for 189261 milliseconds - retrying
2009-05-15 00:55:56,789 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_CALL_SERVER_STARTUP:
safeMode=false
2009-05-15 00:55:57,249 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
On Fri, May 15, 2009 at 7:32 PM, Sasha Dolgy <sd...@gmail.com> wrote:
> Ok, i'll go take a look. They are both on the local server so network
> issues shouldn't be a cause. Cheers though, i'll go look at the JIRA link.
> If I find anything else i'll post here.
> thanks
> -sd
>
> On Fri, May 15, 2009 at 6:18 PM, Andrew Purtell <ap...@apache.org>wrote:
>
>> The region server hosting META could not communicate with the master for a
>> very long time. Some kind of network issue? Any entries in the region server
>> logs above this one
>>
>> > 2009-05-15 00:55:53,090 WARN
>> > org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
>> > master for 189261 milliseconds - retrying
>>
>> which may be relevant? Anything about sleeping too long?
>>
>> Related, there were some bugs that I am aware of preventing recovery if
>> META in particular goes away but they should be fixed for 0.20 as of
>> https://issues.apache.org/jira/browse/HBASE-1362 .
>>
>> - Andy
>>
>
Re: HRegionServer: Failed openScanner
Posted by Sasha Dolgy <sd...@gmail.com>.
In the region server logs I see messages from the 14th:
2009-05-14 22:47:28,840 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region syslog,,1242260881586
2009-05-14 22:47:43,976 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region syslog,,1242260881586 in 15sec
then no log entries until the 15th when the error happens:
2009-05-15 00:55:51,568 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
189138ms, ten times longer than scheduled: 10000
2009-05-15 00:55:52,334 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
188348ms, ten times longer than scheduled: 3000
2009-05-15 00:55:53,090 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
master for 189261 milliseconds - retrying
2009-05-15 00:55:56,789 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_CALL_SERVER_STARTUP:
safeMode=false
2009-05-15 00:55:57,249 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
On Fri, May 15, 2009 at 7:32 PM, Sasha Dolgy <sd...@gmail.com> wrote:
> Ok, i'll go take a look. They are both on the local server so network
> issues shouldn't be a cause. Cheers though, i'll go look at the JIRA link.
> If I find anything else i'll post here.
> thanks
> -sd
>
> On Fri, May 15, 2009 at 6:18 PM, Andrew Purtell <ap...@apache.org>wrote:
>
>> The region server hosting META could not communicate with the master for a
>> very long time. Some kind of network issue? Any entries in the region server
>> logs above this one
>>
>> > 2009-05-15 00:55:53,090 WARN
>> > org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
>> > master for 189261 milliseconds - retrying
>>
>> which may be relevant? Anything about sleeping too long?
>>
>> Related, there were some bugs that I am aware of preventing recovery if
>> META in particular goes away but they should be fixed for 0.20 as of
>> https://issues.apache.org/jira/browse/HBASE-1362 .
>>
>> - Andy
>>
>
Re: HRegionServer: Failed openScanner
Posted by Sasha Dolgy <sd...@gmail.com>.
Ok, i'll go take a look. They are both on the local server so network
issues shouldn't be a cause. Cheers though, i'll go look at the JIRA link.
If I find anything else i'll post here.
thanks
-sd
On Fri, May 15, 2009 at 6:18 PM, Andrew Purtell <ap...@apache.org> wrote:
> The region server hosting META could not communicate with the master for a
> very long time. Some kind of network issue? Any entries in the region server
> logs above this one
>
> > 2009-05-15 00:55:53,090 WARN
> > org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
> > master for 189261 milliseconds - retrying
>
> which may be relevant? Anything about sleeping too long?
>
> Related, there were some bugs that I am aware of preventing recovery if
> META in particular goes away but they should be fixed for 0.20 as of
> https://issues.apache.org/jira/browse/HBASE-1362 .
>
> - Andy
>
Re: HRegionServer: Failed openScanner
Posted by Andrew Purtell <ap...@apache.org>.
The region server hosting META could not communicate with the master for a very long time. Some kind of network issue? Any entries in the region server logs above this one
> 2009-05-15 00:55:53,090 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
> master for 189261 milliseconds - retrying
which may be relevant? Anything about sleeping too long?
Related, there were some bugs that I am aware of preventing recovery if META in particular goes away but they should be fixed for 0.20 as of https://issues.apache.org/jira/browse/HBASE-1362 .
- Andy
________________________________
From: Sasha Dolgy <sd...@gmail.com>
To: hbase-user@hadoop.apache.org
Sent: Friday, May 15, 2009 9:33:24 AM
Subject: Re: HRegionServer: Failed openScanner
on a devel system i don't have any monitoring in place. i just leave it and
hope the log files can give me hints when it breaks. in the master log i
see the log info below. the machine wasn't under any heavy work ... been
the same amount of load for the past 48 hours or so. for the tests
everything is on one machine...
2009-05-15 00:52:49,186 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
startKey: <>,server: 127.0.0.1:60020}
2009-05-15 00:52:49,336 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
startKey: <>,server: 127.0.0.1:60020}
2009-05-15 00:54:43,959 INFO org.apache.hadoop.hbase.master.ServerManager:
127.0.0.1:60020 lease expired
2009-05-15 00:54:47,417 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: process shutdown of
server 127.0.0.1:60020: logSplit: false, rootRescanned: false,
numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
2009-05-15 00:54:50,049 INFO org.apache.hadoop.hbase.regionserver.HLog:
Splitting 27 log(s) in hdfs://
foo.bar.net:9000/hbase/log_127.0.0.1_1242258287822_60020
2009-05-15 00:55:54,866 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of 1 row(s) of meta region {regionname:
-ROOT-,,0, startKey: <>, server: 127.0.0.1:60020} complete
2009-05-15 00:55:57,246 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of 11 row(s) of meta region {regionname:
.META.,,1, startKey: <>, server: 127.0.0.1:60020} complete
2009-05-15 00:55:57,246 INFO org.apache.hadoop.hbase.master.BaseScanner: All
1 .META. region(s) scanned
2009-05-15 00:55:57,246 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
startKey: <>,server: 127.0.0.1:60020}
2009-05-15 00:55:57,727 WARN org.apache.hadoop.hbase.master.BaseScanner:
Scan one META region: {regionname: .META.,,1, startKey: <>, server:
127.0.0.1:60020}
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:185)
at
org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
at
org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
at
org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:137)
at org.apache.hadoop.hbase.Chore.run(Chore.java:65)
2009-05-15 00:55:57,977 INFO org.apache.hadoop.hbase.master.BaseScanner: All
1 .META. region(s) scanned
2009-05-15 00:56:57,252 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
startKey: <>,
server: 127.0.0.1:60020}
2009-05-15 00:57:02,214 WARN org.apache.hadoop.hbase.master.BaseScanner:
Scan one META region: {regionname: .META.,,1, startKey: <>, server:
127.0.0.1:60
020}
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:185)
at
org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
at
org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
On Fri, May 15, 2009 at 5:13 PM, Andrew Purtell <ap...@apache.org> wrote:
> > 2009-05-15 00:55:53,090 WARN
> > org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
> > master for 189261 milliseconds - retrying
>
> What do you see in the master log around this time?
>
> Was your cluster heavily loaded and/or in swap at this time? Do you have
> monitoring in place (i.e. sar (sysstat), Ganglia, Nagios) where you can go
> back in time and look at what was going on at the OS or network level at the
> time?
>
> Best regards,
>
> - Andy
>
>
>
>
> ________________________________
> From: Sasha Dolgy <sa...@gmail.com>
> To: hbase-user@hadoop.apache.org
> Sent: Friday, May 15, 2009 1:53:37 AM
> Subject: HRegionServer: Failed openScanner
>
> Hi there,
>
> Thought I would run a count 'table-name' to see how many records are in a
> table. The count didn't work so I went and took a look in the
> regionserver
> log file and see the below. Now, to be honest, not quite sure why it's
> just
> stopped working. The lines are from the start of the log file (from may
> 15). The log file from May 14 has no errors in it. Is there any way I can
> enable debugging to find out why it's stopped working? I had added this in
> the past 36 hours but it didn't appear to cause any problems and was
> working
> fine for atleast 24 hours:
>
> <property>
> <name>hbase.regionserver.class</name>
> <value>org.apache.hadoop.hbase.ipc.IndexedRegionInterface</value>
> <description>enable indexing</description>
> </property>
>
> <property>
> <name>hbase.regionserver.impl</name>
>
>
> <value>org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegionServer</value>
> <description>enable indexing</description>
> </property>
>
>
> $HBASE_HOME/bin/stop-all.sh and $HBASE_HOME/bin/start-all.sh fixed
> everything and it's all working fine again.
>
> 2009-05-15 00:55:53,090 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
> master for 189261 milliseconds - retrying
> 2009-05-15 00:55:56,789 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_CALL_SERVER_STARTUP:
> safeMode=false
> 2009-05-15 00:55:57,249 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
> 2009-05-15 00:55:57,252 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 9 on 60020, call openScanner([B@233dfbc1, [[B@3a5b4dfa,
> [B@405c7604,
> 9223372036854775807, null) from 127.0.0.1:47277: error:
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
> 2009-05-15 00:56:57,751 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
>
> --
> Sasha Dolgy
> sasha.dolgy@gmail.com
>
>
>
>
>
--
Sasha Dolgy
sasha.dolgy@gmail.com
Re: HRegionServer: Failed openScanner
Posted by Sasha Dolgy <sd...@gmail.com>.
on a devel system i don't have any monitoring in place. i just leave it and
hope the log files can give me hints when it breaks. in the master log i
see the log info below. the machine wasn't under any heavy work ... been
the same amount of load for the past 48 hours or so. for the tests
everything is on one machine...
2009-05-15 00:52:49,186 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
startKey: <>,server: 127.0.0.1:60020}
2009-05-15 00:52:49,336 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
startKey: <>,server: 127.0.0.1:60020}
2009-05-15 00:54:43,959 INFO org.apache.hadoop.hbase.master.ServerManager:
127.0.0.1:60020 lease expired
2009-05-15 00:54:47,417 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: process shutdown of
server 127.0.0.1:60020: logSplit: false, rootRescanned: false,
numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
2009-05-15 00:54:50,049 INFO org.apache.hadoop.hbase.regionserver.HLog:
Splitting 27 log(s) in hdfs://
foo.bar.net:9000/hbase/log_127.0.0.1_1242258287822_60020
2009-05-15 00:55:54,866 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of 1 row(s) of meta region {regionname:
-ROOT-,,0, startKey: <>, server: 127.0.0.1:60020} complete
2009-05-15 00:55:57,246 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of 11 row(s) of meta region {regionname:
.META.,,1, startKey: <>, server: 127.0.0.1:60020} complete
2009-05-15 00:55:57,246 INFO org.apache.hadoop.hbase.master.BaseScanner: All
1 .META. region(s) scanned
2009-05-15 00:55:57,246 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
startKey: <>,server: 127.0.0.1:60020}
2009-05-15 00:55:57,727 WARN org.apache.hadoop.hbase.master.BaseScanner:
Scan one META region: {regionname: .META.,,1, startKey: <>, server:
127.0.0.1:60020}
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:185)
at
org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
at
org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
at
org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:137)
at org.apache.hadoop.hbase.Chore.run(Chore.java:65)
2009-05-15 00:55:57,977 INFO org.apache.hadoop.hbase.master.BaseScanner: All
1 .META. region(s) scanned
2009-05-15 00:56:57,252 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
startKey: <>,
server: 127.0.0.1:60020}
2009-05-15 00:57:02,214 WARN org.apache.hadoop.hbase.master.BaseScanner:
Scan one META region: {regionname: .META.,,1, startKey: <>, server:
127.0.0.1:60
020}
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:185)
at
org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
at
org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
On Fri, May 15, 2009 at 5:13 PM, Andrew Purtell <ap...@apache.org> wrote:
> > 2009-05-15 00:55:53,090 WARN
> > org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
> > master for 189261 milliseconds - retrying
>
> What do you see in the master log around this time?
>
> Was your cluster heavily loaded and/or in swap at this time? Do you have
> monitoring in place (i.e. sar (sysstat), Ganglia, Nagios) where you can go
> back in time and look at what was going on at the OS or network level at the
> time?
>
> Best regards,
>
> - Andy
>
>
>
>
> ________________________________
> From: Sasha Dolgy <sa...@gmail.com>
> To: hbase-user@hadoop.apache.org
> Sent: Friday, May 15, 2009 1:53:37 AM
> Subject: HRegionServer: Failed openScanner
>
> Hi there,
>
> Thought I would run a count 'table-name' to see how many records are in a
> table. The count didn't work so I went and took a look in the
> regionserver
> log file and see the below. Now, to be honest, not quite sure why it's
> just
> stopped working. The lines are from the start of the log file (from may
> 15). The log file from May 14 has no errors in it. Is there any way I can
> enable debugging to find out why it's stopped working? I had added this in
> the past 36 hours but it didn't appear to cause any problems and was
> working
> fine for atleast 24 hours:
>
> <property>
> <name>hbase.regionserver.class</name>
> <value>org.apache.hadoop.hbase.ipc.IndexedRegionInterface</value>
> <description>enable indexing</description>
> </property>
>
> <property>
> <name>hbase.regionserver.impl</name>
>
>
> <value>org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegionServer</value>
> <description>enable indexing</description>
> </property>
>
>
> $HBASE_HOME/bin/stop-all.sh and $HBASE_HOME/bin/start-all.sh fixed
> everything and it's all working fine again.
>
> 2009-05-15 00:55:53,090 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
> master for 189261 milliseconds - retrying
> 2009-05-15 00:55:56,789 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_CALL_SERVER_STARTUP:
> safeMode=false
> 2009-05-15 00:55:57,249 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
> 2009-05-15 00:55:57,252 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 9 on 60020, call openScanner([B@233dfbc1, [[B@3a5b4dfa,
> [B@405c7604,
> 9223372036854775807, null) from 127.0.0.1:47277: error:
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
> 2009-05-15 00:56:57,751 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
>
> --
> Sasha Dolgy
> sasha.dolgy@gmail.com
>
>
>
>
>
--
Sasha Dolgy
sasha.dolgy@gmail.com
Re: HRegionServer: Failed openScanner
Posted by Andrew Purtell <ap...@apache.org>.
> 2009-05-15 00:55:53,090 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
> master for 189261 milliseconds - retrying
What do you see in the master log around this time?
Was your cluster heavily loaded and/or in swap at this time? Do you have monitoring in place (i.e. sar (sysstat), Ganglia, Nagios) where you can go back in time and look at what was going on at the OS or network level at the time?
Best regards,
- Andy
________________________________
From: Sasha Dolgy <sa...@gmail.com>
To: hbase-user@hadoop.apache.org
Sent: Friday, May 15, 2009 1:53:37 AM
Subject: HRegionServer: Failed openScanner
Hi there,
Thought I would run a count 'table-name' to see how many records are in a
table. The count didn't work so I went and took a look in the regionserver
log file and see the below. Now, to be honest, not quite sure why it's just
stopped working. The lines are from the start of the log file (from may
15). The log file from May 14 has no errors in it. Is there any way I can
enable debugging to find out why it's stopped working? I had added this in
the past 36 hours but it didn't appear to cause any problems and was working
fine for atleast 24 hours:
<property>
<name>hbase.regionserver.class</name>
<value>org.apache.hadoop.hbase.ipc.IndexedRegionInterface</value>
<description>enable indexing</description>
</property>
<property>
<name>hbase.regionserver.impl</name>
<value>org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegionServer</value>
<description>enable indexing</description>
</property>
$HBASE_HOME/bin/stop-all.sh and $HBASE_HOME/bin/start-all.sh fixed
everything and it's all working fine again.
2009-05-15 00:55:53,090 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
master for 189261 milliseconds - retrying
2009-05-15 00:55:56,789 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_CALL_SERVER_STARTUP:
safeMode=false
2009-05-15 00:55:57,249 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
2009-05-15 00:55:57,252 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60020, call openScanner([B@233dfbc1, [[B@3a5b4dfa, [B@405c7604,
9223372036854775807, null) from 127.0.0.1:47277: error:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
2009-05-15 00:56:57,751 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
--
Sasha Dolgy
sasha.dolgy@gmail.com