got HBASE-2516 on 0.20.6

[Andrey Stepachev <oc...@gmail.com>]

I need to massive data rewrite in some family on standalone server. I
got org.apache.hadoop.hbase.NotServingRegionException
or java.io.IOException: Region xxx closed if I write and read at the same time.

What i shall do in 0.20.6 version? One thing that i can try - write to
another family and then copy changed data
back to original family. But this is not really good solution.
Another option - got to 0.89, but i don't know where it will be
released, so it is very questionable, is it really usable
(small standalone server with up to 5 nodes in the future).

Re: got HBASE-2516 on 0.20.6

[Andrey Stepachev <oc...@gmail.com>]

Thanks for comment.

2010/9/18 Jean-Daniel Cryans <jd...@apache.org>:
> That exception is "normal", it just means a region is either moving or
> being split, unless your client has to retry 10 times and then gets a
> RetriesExhaustedException. This has been discussed a few times on this
> mailing list.

I have custom RetryResultIterator, which reopen scanner from last known
postition.

>
> Also you say you got HBASE-2516, but if you really got it then you'd
> see IOException instead of NotServingRegionException.

I got it in different situations.

>
> Are you actually having issues importing your data? If so, can you be
> more specific about how it fails?

I have several families and 70% of data should be changed (change in
json object). I open scanner and go through, make new changed puts
(they are placed in same key/family/qualifier) and put them in the same
table/family.

I wrote a very small tool (looks like mr job, but simply java app)
which spawns serveral threads,
gets key ranges and each thread works with only one family at a time
and one key range.
Tool do something like this (executor pool has fixed number of threads):

families.each { |f|
    keyranges.each { |range|
        executorPool.startThread(f, range)
    }
}

If I run tool on two familes, and looks like region splits on first
family, when i process second family, i got
http://pastebin.com/M4xzaHAg.
If I start tool only with failed family, i got "region closed".
http://pastebin.com/9zFLi9Pn
I really can't figure out what is going on. But I think that here is
some "race" when region begins
to split.

>
> Regarding 0.89, since you are running HBase on only one machine, I
> don't expect that you'll run into any major issues. Here at
> StumbleUpon we have been running on 0.89.20100830 (the release
> candidates) since the first week of september in our production
> environment.

Thanks. Interesting. I'll try to move my test server and look how it works.
0.89 has too many features (like timestamp scan) that i need.

>
> J-D
>
> On Fri, Sep 17, 2010 at 1:36 PM, Andrey Stepachev <oc...@gmail.com> wrote:
>> I need to massive data rewrite in some family on standalone server. I
>> got org.apache.hadoop.hbase.NotServingRegionException
>> or java.io.IOException: Region xxx closed if I write and read at the same time.
>>
>> What i shall do in 0.20.6 version? One thing that i can try - write to
>> another family and then copy changed data
>> back to original family. But this is not really good solution.
>> Another option - got to 0.89, but i don't know where it will be
>> released, so it is very questionable, is it really usable
>> (small standalone server with up to 5 nodes in the future).
>>
>

Re: got HBASE-2516 on 0.20.6

[Jean-Daniel Cryans <jd...@apache.org>]

That exception is "normal", it just means a region is either moving or
being split, unless your client has to retry 10 times and then gets a
RetriesExhaustedException. This has been discussed a few times on this
mailing list.

Also you say you got HBASE-2516, but if you really got it then you'd
see IOException instead of NotServingRegionException.

Are you actually having issues importing your data? If so, can you be
more specific about how it fails?

Regarding 0.89, since you are running HBase on only one machine, I
don't expect that you'll run into any major issues. Here at
StumbleUpon we have been running on 0.89.20100830 (the release
candidates) since the first week of september in our production
environment.

J-D

On Fri, Sep 17, 2010 at 1:36 PM, Andrey Stepachev <oc...@gmail.com> wrote:
> I need to massive data rewrite in some family on standalone server. I
> got org.apache.hadoop.hbase.NotServingRegionException
> or java.io.IOException: Region xxx closed if I write and read at the same time.
>
> What i shall do in 0.20.6 version? One thing that i can try - write to
> another family and then copy changed data
> back to original family. But this is not really good solution.
> Another option - got to 0.89, but i don't know where it will be
> released, so it is very questionable, is it really usable
> (small standalone server with up to 5 nodes in the future).
>