online_merge on 0.92.1 cdh4

[Erdem Agaoglu <er...@gmail.com>]

Hi all,

We tried online_merge on one of our tables today and got screwed big time.
dryrun failed several times because unknown constructors of HServerAddress,
HRegionInfo.

We fixed those and passed to normalRun and failed on admin.closeRegion
because online_merge was providing region server as 'host:port' while
method expects it as 'host,port,startsecond'.
StringIndexOutOfBoundsException was pretty helpful.

We fixed that too, only to fail at admin.assign because method was not
expecting a second argument, which was set to true at online_merge.rb. We
think we did fix that too, but it was no longer relevant because we had a
table with unassigned regions and a broken .META. record. Applications were
warning with

o.a.h.h.c.HConnectionManager$HConnectionImplementation - Encountered
problems when prefetch META table:
java.io.IOException: HRegionInfo was null or empty in Meta for <tablename>

We discovered that there were rows in .META. about the table without
info:regioninfo column and found out that these were causing warnings, so
deleted those rows. This fixed these warnings.

After that we saw that the only region of the table after the merge has an
endKey attribute, so our scan's or get's fail with WrongRegionException. We
tried to change the endKey value in .META. but we killed .META. scanning in
the process. Deleted the tampered row and hbck -repair'ed only to get back
to square one: single region with an endKey.

Any ideas on how to set a region definition without an endKey would be
greatly appreciated.

-- 
erdem agaoglu

Re: online_merge on 0.92.1 cdh4

[Erdem Agaoglu <er...@gmail.com>]

Hi again,

Defining a new region with no start/end key and moving the existing one to
this newly defined seems to fixed the problem. For interested folks here is
what we did:

On shell
c = HBaseConfiguration.create()
table = HTable.new(c, "mytable")
meta = HTable.new(c, HConstants::META_TABLE_NAME)
tableDesc = table.getTableDescriptor

newHRI = HRegionInfo.new(tableDesc.getName)
put = Put.new(newHRI.getRegionName)
put.add(HConstants::CATALOG_FAMILY, HConstants::REGIONINFO_QUALIFIER,
Writables.getBytes(newHRI))
meta.put(put)

At this point we had two rows for mytable in .META., one with an endKey and
one with none. We deleted the row with endKey and renamed the region to
match the newly generated row in HDFS. Last thing to do was simply assign
the region to a RegionServer.

On Wed, Oct 3, 2012 at 5:59 PM, Erdem Agaoglu <er...@gmail.com>wrote:

> Hi all,
>
> We tried online_merge on one of our tables today and got screwed big time.
> dryrun failed several times because unknown constructors of HServerAddress,
> HRegionInfo.
>
> We fixed those and passed to normalRun and failed on admin.closeRegion
> because online_merge was providing region server as 'host:port' while
> method expects it as 'host,port,startsecond'.
> StringIndexOutOfBoundsException was pretty helpful.
>
> We fixed that too, only to fail at admin.assign because method was not
> expecting a second argument, which was set to true at online_merge.rb. We
> think we did fix that too, but it was no longer relevant because we had a
> table with unassigned regions and a broken .META. record. Applications were
> warning with
>
> o.a.h.h.c.HConnectionManager$HConnectionImplementation - Encountered
> problems when prefetch META table:
> java.io.IOException: HRegionInfo was null or empty in Meta for <tablename>
>
> We discovered that there were rows in .META. about the table without
> info:regioninfo column and found out that these were causing warnings, so
> deleted those rows. This fixed these warnings.
>
> After that we saw that the only region of the table after the merge has an
> endKey attribute, so our scan's or get's fail with WrongRegionException. We
> tried to change the endKey value in .META. but we killed .META. scanning in
> the process. Deleted the tampered row and hbck -repair'ed only to get back
> to square one: single region with an endKey.
>
> Any ideas on how to set a region definition without an endKey would be
> greatly appreciated.
>
> --
> erdem agaoglu
>



-- 
erdem agaoglu