You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Vinod Gupta Tankala <tv...@socialyantra.com> on 2011/09/27 16:57:12 UTC
Java client throws WrongRegionException but same key accessible via
hbase shell
I find this hard to believe. For the same row key, my java client is
throwing wrong region exception. But I can query the same using hbase shell.
Im on 0.90.2 version.
Also note that I have inconsistencies in my regions that I am still trying
to figure out. But regardless, the inconsistencies should impact both
methods of querying similarly. right?
thanks
vinod
Re: Java client throws WrongRegionException but same key accessible
via hbase shell
Posted by Rohit Nigam <ta...@gmail.com>.
Try running
hbase org.jruby.Main add_table.rb /hbase/tablename
This will clean the inconsistencies in .META. table . If you see run hbck
again and you see holes in the table then you have to do more effort in
cleaning the table.
Rohit
On Tue, Sep 27, 2011 at 1:23 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> Yes, please send the info I asked.
>
> About the hbck errors you had, this is usually fixed with -fix:
>
> Region
> AkStats,277808094:1314921600:daily:Volume,1317052667861.1ecc871503cd827934c3a9077b44e52b.
> not deployed on any region server.
>
> This is "probably" a region that wasn't cleaned up so it's not really a
> problem:
>
> Region
> file:/media/ephemeral0/hbase-data/AkStats/3520c0cfbedcf212084379c0e41e7839
> on HDFS, but not listed in META or deployed on any region server.
>
> This is a real problem:
>
> Chain of regions in table AkStats is broken; edges does not contain
> 277808094:1314921600:daily:Volume
>
> J-D
>
> On Tue, Sep 27, 2011 at 1:00 PM, Vinod Gupta Tankala
> <tv...@readypulse.com> wrote:
> > J-D,
> > I was getting these errors even after restarting the client. So it
> probably
> > is not straightforward.
> > Also, I was able to run a combination of check_meta.rb and hbck with
> their
> > fix options and restore some of the inconsistencies. i still have 4
> > inconsistencies left (earlier it was 7) but check_meta thinks .META. is
> > fine. After i did this, I am not getting java client errors any more. Not
> > sure how to explain that.
> >
> > Do you still want me to send the information you requested? may be you
> can
> > help with remaining inconsistencies.
> >
> > thanks
> > vinod
> >
> > On Tue, Sep 27, 2011 at 11:05 AM, Jean-Daniel Cryans <
> jdcryans@apache.org>wrote:
> >
> >> Hi vinod,
> >>
> >> Yeah WREs are never fun, hopefully we can help you fixing it.
> >>
> >> First, about the difference when querying from the shell and your java
> >> client.
> >>
> >> - Is it a long lived client? Did you restart it since you got the WREs?
> >> - If not, this could just be due to the fact that it has a cache of
> >> regions that's different from what a new client would see.
> >> - If you did restart it, then I would have to think a bit more about
> >> it to find the difference.
> >>
> >> Either way, it'd be nice to see what you're doing. We need;
> >>
> >> - A full dump of your .META. (in the shell: scan '.META.'), please
> >> put this on a web server or a pastebin
> >> - The keys you are trying to reach
> >> - The exceptions you are seeing that contain the row keys and regions
> >> it's trying to reach
> >> - Another thing that would be nice is to have the output of when you
> >> are reaching that row key from the shell, but with the shell started
> >> with the -d option (will show a lot more debug info).
> >> - The master log of the day the exception started happening.
> >>
> >> J-D
> >>
> >> On Tue, Sep 27, 2011 at 7:57 AM, Vinod Gupta Tankala
> >> <tv...@socialyantra.com> wrote:
> >> > I find this hard to believe. For the same row key, my java client is
> >> > throwing wrong region exception. But I can query the same using hbase
> >> shell.
> >> > Im on 0.90.2 version.
> >> >
> >> > Also note that I have inconsistencies in my regions that I am still
> >> trying
> >> > to figure out. But regardless, the inconsistencies should impact both
> >> > methods of querying similarly. right?
> >> >
> >> > thanks
> >> > vinod
> >> >
> >>
> >
>
Re: Java client throws WrongRegionException but same key accessible
via hbase shell
Posted by Vinod Gupta Tankala <tv...@readypulse.com>.
J-D,
here is the meta scan file -
https://docs.google.com/document/d/1_g2Ce20H65rukrEe8i9UWaW9wskaV34_96ElpOQDsYE/edit?hl=en_US
I solved this problem "Chain of regions in table AkStats is broken; edges
does not contain
277808094:1314921600:daily:Volume" using combination of check_meta and hbck
-fix.
But every now and then i hit this problem where java client hits WRE but
hbase shell works.
an example of the above is -
get 'AkStats', '26696569976:1317081600:weekly:AudEng'
I have a suspicion on why this could be happening. Please confirm if I
should be concerned about the following -
1) 5-10% of my rows are really large. ~2.5MB+. Remaining are smaller - few
KB.
2) Before I write the large rows, I delete the row if it exists and then
write a new one. The reason I do it this way is because some of the columns
in the existing row don't apply any more. So I delete the whole row and
rewrite it. Ofcourse, this happens every few hours for 1-2K rows only based
on my current load.
Does hbase scale well for row deletes?
thanks
On Tue, Sep 27, 2011 at 1:23 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> Yes, please send the info I asked.
>
> About the hbck errors you had, this is usually fixed with -fix:
>
> Region
> AkStats,277808094:1314921600:daily:Volume,1317052667861.1ecc871503cd827934c3a9077b44e52b.
> not deployed on any region server.
>
> This is "probably" a region that wasn't cleaned up so it's not really a
> problem:
>
> Region
> file:/media/ephemeral0/hbase-data/AkStats/3520c0cfbedcf212084379c0e41e7839
> on HDFS, but not listed in META or deployed on any region server.
>
> This is a real problem:
>
> Chain of regions in table AkStats is broken; edges does not contain
> 277808094:1314921600:daily:Volume
>
> J-D
>
> On Tue, Sep 27, 2011 at 1:00 PM, Vinod Gupta Tankala
> <tv...@readypulse.com> wrote:
> > J-D,
> > I was getting these errors even after restarting the client. So it
> probably
> > is not straightforward.
> > Also, I was able to run a combination of check_meta.rb and hbck with
> their
> > fix options and restore some of the inconsistencies. i still have 4
> > inconsistencies left (earlier it was 7) but check_meta thinks .META. is
> > fine. After i did this, I am not getting java client errors any more. Not
> > sure how to explain that.
> >
> > Do you still want me to send the information you requested? may be you
> can
> > help with remaining inconsistencies.
> >
> > thanks
> > vinod
> >
> > On Tue, Sep 27, 2011 at 11:05 AM, Jean-Daniel Cryans <
> jdcryans@apache.org>wrote:
> >
> >> Hi vinod,
> >>
> >> Yeah WREs are never fun, hopefully we can help you fixing it.
> >>
> >> First, about the difference when querying from the shell and your java
> >> client.
> >>
> >> - Is it a long lived client? Did you restart it since you got the WREs?
> >> - If not, this could just be due to the fact that it has a cache of
> >> regions that's different from what a new client would see.
> >> - If you did restart it, then I would have to think a bit more about
> >> it to find the difference.
> >>
> >> Either way, it'd be nice to see what you're doing. We need;
> >>
> >> - A full dump of your .META. (in the shell: scan '.META.'), please
> >> put this on a web server or a pastebin
> >> - The keys you are trying to reach
> >> - The exceptions you are seeing that contain the row keys and regions
> >> it's trying to reach
> >> - Another thing that would be nice is to have the output of when you
> >> are reaching that row key from the shell, but with the shell started
> >> with the -d option (will show a lot more debug info).
> >> - The master log of the day the exception started happening.
> >>
> >> J-D
> >>
> >> On Tue, Sep 27, 2011 at 7:57 AM, Vinod Gupta Tankala
> >> <tv...@socialyantra.com> wrote:
> >> > I find this hard to believe. For the same row key, my java client is
> >> > throwing wrong region exception. But I can query the same using hbase
> >> shell.
> >> > Im on 0.90.2 version.
> >> >
> >> > Also note that I have inconsistencies in my regions that I am still
> >> trying
> >> > to figure out. But regardless, the inconsistencies should impact both
> >> > methods of querying similarly. right?
> >> >
> >> > thanks
> >> > vinod
> >> >
> >>
> >
>
Re: Java client throws WrongRegionException but same key accessible
via hbase shell
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Yes, please send the info I asked.
About the hbck errors you had, this is usually fixed with -fix:
Region AkStats,277808094:1314921600:daily:Volume,1317052667861.1ecc871503cd827934c3a9077b44e52b.
not deployed on any region server.
This is "probably" a region that wasn't cleaned up so it's not really a problem:
Region file:/media/ephemeral0/hbase-data/AkStats/3520c0cfbedcf212084379c0e41e7839
on HDFS, but not listed in META or deployed on any region server.
This is a real problem:
Chain of regions in table AkStats is broken; edges does not contain
277808094:1314921600:daily:Volume
J-D
On Tue, Sep 27, 2011 at 1:00 PM, Vinod Gupta Tankala
<tv...@readypulse.com> wrote:
> J-D,
> I was getting these errors even after restarting the client. So it probably
> is not straightforward.
> Also, I was able to run a combination of check_meta.rb and hbck with their
> fix options and restore some of the inconsistencies. i still have 4
> inconsistencies left (earlier it was 7) but check_meta thinks .META. is
> fine. After i did this, I am not getting java client errors any more. Not
> sure how to explain that.
>
> Do you still want me to send the information you requested? may be you can
> help with remaining inconsistencies.
>
> thanks
> vinod
>
> On Tue, Sep 27, 2011 at 11:05 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> Hi vinod,
>>
>> Yeah WREs are never fun, hopefully we can help you fixing it.
>>
>> First, about the difference when querying from the shell and your java
>> client.
>>
>> - Is it a long lived client? Did you restart it since you got the WREs?
>> - If not, this could just be due to the fact that it has a cache of
>> regions that's different from what a new client would see.
>> - If you did restart it, then I would have to think a bit more about
>> it to find the difference.
>>
>> Either way, it'd be nice to see what you're doing. We need;
>>
>> - A full dump of your .META. (in the shell: scan '.META.'), please
>> put this on a web server or a pastebin
>> - The keys you are trying to reach
>> - The exceptions you are seeing that contain the row keys and regions
>> it's trying to reach
>> - Another thing that would be nice is to have the output of when you
>> are reaching that row key from the shell, but with the shell started
>> with the -d option (will show a lot more debug info).
>> - The master log of the day the exception started happening.
>>
>> J-D
>>
>> On Tue, Sep 27, 2011 at 7:57 AM, Vinod Gupta Tankala
>> <tv...@socialyantra.com> wrote:
>> > I find this hard to believe. For the same row key, my java client is
>> > throwing wrong region exception. But I can query the same using hbase
>> shell.
>> > Im on 0.90.2 version.
>> >
>> > Also note that I have inconsistencies in my regions that I am still
>> trying
>> > to figure out. But regardless, the inconsistencies should impact both
>> > methods of querying similarly. right?
>> >
>> > thanks
>> > vinod
>> >
>>
>
Re: Java client throws WrongRegionException but same key accessible
via hbase shell
Posted by Vinod Gupta Tankala <tv...@readypulse.com>.
J-D,
I was getting these errors even after restarting the client. So it probably
is not straightforward.
Also, I was able to run a combination of check_meta.rb and hbck with their
fix options and restore some of the inconsistencies. i still have 4
inconsistencies left (earlier it was 7) but check_meta thinks .META. is
fine. After i did this, I am not getting java client errors any more. Not
sure how to explain that.
Do you still want me to send the information you requested? may be you can
help with remaining inconsistencies.
thanks
vinod
On Tue, Sep 27, 2011 at 11:05 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> Hi vinod,
>
> Yeah WREs are never fun, hopefully we can help you fixing it.
>
> First, about the difference when querying from the shell and your java
> client.
>
> - Is it a long lived client? Did you restart it since you got the WREs?
> - If not, this could just be due to the fact that it has a cache of
> regions that's different from what a new client would see.
> - If you did restart it, then I would have to think a bit more about
> it to find the difference.
>
> Either way, it'd be nice to see what you're doing. We need;
>
> - A full dump of your .META. (in the shell: scan '.META.'), please
> put this on a web server or a pastebin
> - The keys you are trying to reach
> - The exceptions you are seeing that contain the row keys and regions
> it's trying to reach
> - Another thing that would be nice is to have the output of when you
> are reaching that row key from the shell, but with the shell started
> with the -d option (will show a lot more debug info).
> - The master log of the day the exception started happening.
>
> J-D
>
> On Tue, Sep 27, 2011 at 7:57 AM, Vinod Gupta Tankala
> <tv...@socialyantra.com> wrote:
> > I find this hard to believe. For the same row key, my java client is
> > throwing wrong region exception. But I can query the same using hbase
> shell.
> > Im on 0.90.2 version.
> >
> > Also note that I have inconsistencies in my regions that I am still
> trying
> > to figure out. But regardless, the inconsistencies should impact both
> > methods of querying similarly. right?
> >
> > thanks
> > vinod
> >
>
Re: Java client throws WrongRegionException but same key accessible
via hbase shell
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Hi vinod,
Yeah WREs are never fun, hopefully we can help you fixing it.
First, about the difference when querying from the shell and your java client.
- Is it a long lived client? Did you restart it since you got the WREs?
- If not, this could just be due to the fact that it has a cache of
regions that's different from what a new client would see.
- If you did restart it, then I would have to think a bit more about
it to find the difference.
Either way, it'd be nice to see what you're doing. We need;
- A full dump of your .META. (in the shell: scan '.META.'), please
put this on a web server or a pastebin
- The keys you are trying to reach
- The exceptions you are seeing that contain the row keys and regions
it's trying to reach
- Another thing that would be nice is to have the output of when you
are reaching that row key from the shell, but with the shell started
with the -d option (will show a lot more debug info).
- The master log of the day the exception started happening.
J-D
On Tue, Sep 27, 2011 at 7:57 AM, Vinod Gupta Tankala
<tv...@socialyantra.com> wrote:
> I find this hard to believe. For the same row key, my java client is
> throwing wrong region exception. But I can query the same using hbase shell.
> Im on 0.90.2 version.
>
> Also note that I have inconsistencies in my regions that I am still trying
> to figure out. But regardless, the inconsistencies should impact both
> methods of querying similarly. right?
>
> thanks
> vinod
>