You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Parmod Mehta <pa...@gmail.com> on 2011/09/12 19:33:35 UTC

HBase region merge problem

Running the merge tool to merge regions on hbase-0.90.1-cdh3u0 run into this
exception trace. The first INFO level log message I guess is ok (checking if
the cluster is down).

Not sure - why we running into this? bug?

java.lang.IllegalArgumentException: Wrong FS: hdfs://
internalhost.com:54310/hbase/-ROOT-/70236052/.regioninfo, expected: file:///


2011-09-12 10:21:38,364 INFO  [main] util.Merge(82): Verifying that file
system is available...
2011-09-12 10:21:38,370 INFO  [main] util.Merge(91): Verifying that HBase is
not running...
2011-09-12 10:21:38,576 INFO  [main]
client.HConnectionManager$HConnectionImplementation(355): ZooKeeper
available but no active master location found
2011-09-12 10:21:38,578 INFO  [main]
client.HConnectionManager$HConnectionImplementation(372): getMaster attempt
0 of 1 failed; no more retrying.
org.apache.hadoop.hbase.MasterNotRunningException
        at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:356)
        at
org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:90)
        at
org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:1202)
        at org.apache.hadoop.hbase.util.Merge.run(Merge.java:93)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.hbase.util.Merge.main(Merge.java:379)
2011-09-12 10:21:38,913 INFO  [main] util.Merge(199): Merging regions
TABLE_2,ERSx0973541,1305737576487.c0a59ee581f619456c962227073f9dd6. and
TABLE_2,LEx09136249,1305737576487.85f0adaeb78d362596ea8d178bda5629. in table
TABLE_2
2011-09-12 10:21:38,940 INFO  [main] wal.HLog(364): HLog configuration:
blocksize=32 MB, rollsize=30.4 MB, enabled=true, flushlogentries=1,
optionallogflushinternal=1000ms
2011-09-12 10:21:39,016 WARN  [main] util.NativeCodeLoader(52): Unable to
load native-hadoop library for your platform... using builtin-java classes
where applicable
2011-09-12 10:21:39,602 INFO  [main] wal.SequenceFileLogWriter(126): Using
syncFs -- HDFS-200
2011-09-12 10:21:39,605 INFO  [main] wal.HLog(510): New hlog
/home/pmehta/.logs_1315848098931/hlog.1315848098940
2011-09-12 10:21:39,606 INFO  [main] wal.HLog(396):
getNumCurrentReplicas--HDFS-826 not available;
hdfs_out=org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer@3a289d2e,
exception=org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.getNumCurrentReplicas()
2011-09-12 10:21:39,631 FATAL [main] util.Merge(114): Merge failed
java.lang.IllegalArgumentException: Wrong FS: hdfs://
internalhost.com:54310/hbase/-ROOT-/70236052/.regioninfo, expected: file:///
        at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:410)
        at
org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:56)
        at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:379)
        at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:748)
        at
org.apache.hadoop.hbase.regionserver.HRegion.checkRegioninfoOnFilesystem(HRegion.java:408)
        at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:337)
        at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2546)
        at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2532)
        at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2501)
        at
org.apache.hadoop.hbase.util.MetaUtils.openRootRegion(MetaUtils.java:269)
        at
org.apache.hadoop.hbase.util.MetaUtils.scanRootRegion(MetaUtils.java:201)
        at
org.apache.hadoop.hbase.util.Merge.mergeTwoRegions(Merge.java:204)
        at org.apache.hadoop.hbase.util.Merge.run(Merge.java:110)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.hbase.util.Merge.main(Merge.java:379)
2011-09-12 10:21:39,634 INFO  [main.logSyncer] wal.HLog$LogSyncer(973):
main.logSyncer exiting
2011-09-12 10:21:39,640 ERROR [main] util.Merge(381): exiting due to error
java.lang.IllegalArgumentException: Wrong FS: hdfs://
internalhost.com:54310/hbase/.META./1028785192/.regioninfo, expected:
file:///
        at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:410)
        at
org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:56)
        at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:379)
        at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:748)
        at
org.apache.hadoop.hbase.regionserver.HRegion.checkRegioninfoOnFilesystem(HRegion.java:408)
        at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:337)
        at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2546)
        at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2532)
        at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2501)
        at
org.apache.hadoop.hbase.util.MetaUtils.openMetaRegion(MetaUtils.java:276)
        at
org.apache.hadoop.hbase.util.MetaUtils.scanMetaRegion(MetaUtils.java:261)
        at org.apache.hadoop.hbase.util.Merge.run(Merge.java:115)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.hbase.util.Merge.main(Merge.java:379)


thanks

Re: HBase region merge problem

Posted by Stack <st...@duboce.net>.
I see this in your first set of errors:

2011-09-12 10:21:39,606 INFO  [main] wal.HLog(396):
getNumCurrentReplicas--HDFS-826 not available;

That seems strange for a CDH -- IIRC, it should have the above.

What is the context you are running the merge in?  My guess is that we
are finding other than hbase-site.xml with fs.default.name first when
Configuration picks up config. files.  Are you running for mapreduce
context?  (Or from a webcontainer context)?

St.Ack

On Mon, Oct 17, 2011 at 3:20 PM, Parmod Mehta <pa...@gmail.com> wrote:
> Thanks J-D, the problem was resolved in our dev cluster but have reappeared
> in our staging cluster. I added the fs.default.name in the hbase-site.xml as
> I did it in the dev cluster with no luck (we running hbase-0.90.4)
>
>    [exec] java.lang.IllegalArgumentException: Wrong FS:
> file:/tmp/hbase-root/hbase/-ROOT-/70236052/.regioninfo, expected: hdfs://
> staging.server.com:54310
>     [exec]     at
> org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:410)
>     [exec]     at
> org.apache.hadoop.hdfs.DistributedFileSystem.checkPath(DistributedFileSystem.java:106)
>     [exec]     at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:162)
>     [exec]     at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:542)
>     [exec]     at
> org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:748)
>     [exec]     at
> org.apache.hadoop.hbase.regionserver.HRegion.checkRegioninfoOnFilesystem(HRegion.java:413)
>     [exec]     at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:342)
>     [exec]     at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2564)
>     [exec]     at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2550)
>     [exec]     at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2519)
>     [exec]     at
> org.apache.hadoop.hbase.util.MetaUtils.openRootRegion(MetaUtils.java:269)
>     [exec]     at
> org.apache.hadoop.hbase.util.MetaUtils.scanRootRegion(MetaUtils.java:201)
>     [exec]     at
> org.apache.hadoop.hbase.util.Merge.mergeTwoRegions(Merge.java:204)
>     [exec]     at org.apache.hadoop.hbase.util.Merge.run(Merge.java:110)
>     [exec]     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>     [exec]     at
> com.telcontar.tools.data.util.MergeUtil.runMergeTool(MergeUtil.java:57)
>     [exec]     at
> com.telcontar.tools.data.util.MergeUtil.run(MergeUtil.java:120)
>     [exec]     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>     [exec]     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>     [exec]     at
> com.telcontar.tools.data.util.MergeUtil.main(MergeUtil.java:100)
>     [exec]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>     [exec]     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>     [exec]     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     [exec]     at java.lang.reflect.Method.invoke(Method.java:616)
>     [exec]     at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
>     [exec] 11/10/17 15:16:33 INFO wal.HLog: main.logSyncer exiting
>
>
>
>
> On Mon, Sep 12, 2011 at 10:45 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> Usually it means you need to set fs.default.name in hbase-site.xml
>>
>> J-D
>>
>> On Mon, Sep 12, 2011 at 10:33 AM, Parmod Mehta <pa...@gmail.com>
>> wrote:
>> > Running the merge tool to merge regions on hbase-0.90.1-cdh3u0 run into
>> this
>> > exception trace. The first INFO level log message I guess is ok (checking
>> if
>> > the cluster is down).
>> >
>> > Not sure - why we running into this? bug?
>> >
>> > java.lang.IllegalArgumentException: Wrong FS: hdfs://
>> > internalhost.com:54310/hbase/-ROOT-/70236052/.regioninfo, expected:
>> file:///
>> >
>> >
>> > 2011-09-12 10:21:38,364 INFO  [main] util.Merge(82): Verifying that file
>> > system is available...
>> > 2011-09-12 10:21:38,370 INFO  [main] util.Merge(91): Verifying that HBase
>> is
>> > not running...
>> > 2011-09-12 10:21:38,576 INFO  [main]
>> > client.HConnectionManager$HConnectionImplementation(355): ZooKeeper
>> > available but no active master location found
>> > 2011-09-12 10:21:38,578 INFO  [main]
>> > client.HConnectionManager$HConnectionImplementation(372): getMaster
>> attempt
>> > 0 of 1 failed; no more retrying.
>> > org.apache.hadoop.hbase.MasterNotRunningException
>> >        at
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:356)
>> >        at
>> > org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:90)
>> >        at
>> >
>> org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:1202)
>> >        at org.apache.hadoop.hbase.util.Merge.run(Merge.java:93)
>> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>> >        at org.apache.hadoop.hbase.util.Merge.main(Merge.java:379)
>> > 2011-09-12 10:21:38,913 INFO  [main] util.Merge(199): Merging regions
>> > TABLE_2,ERSx0973541,1305737576487.c0a59ee581f619456c962227073f9dd6. and
>> > TABLE_2,LEx09136249,1305737576487.85f0adaeb78d362596ea8d178bda5629. in
>> table
>> > TABLE_2
>> > 2011-09-12 10:21:38,940 INFO  [main] wal.HLog(364): HLog configuration:
>> > blocksize=32 MB, rollsize=30.4 MB, enabled=true, flushlogentries=1,
>> > optionallogflushinternal=1000ms
>> > 2011-09-12 10:21:39,016 WARN  [main] util.NativeCodeLoader(52): Unable to
>> > load native-hadoop library for your platform... using builtin-java
>> classes
>> > where applicable
>> > 2011-09-12 10:21:39,602 INFO  [main] wal.SequenceFileLogWriter(126):
>> Using
>> > syncFs -- HDFS-200
>> > 2011-09-12 10:21:39,605 INFO  [main] wal.HLog(510): New hlog
>> > /home/pmehta/.logs_1315848098931/hlog.1315848098940
>> > 2011-09-12 10:21:39,606 INFO  [main] wal.HLog(396):
>> > getNumCurrentReplicas--HDFS-826 not available;
>> >
>> hdfs_out=org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer@3a289d2e
>> ,
>> >
>> exception=org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.getNumCurrentReplicas()
>> > 2011-09-12 10:21:39,631 FATAL [main] util.Merge(114): Merge failed
>> > java.lang.IllegalArgumentException: Wrong FS: hdfs://
>> > internalhost.com:54310/hbase/-ROOT-/70236052/.regioninfo, expected:
>> file:///
>> >        at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:410)
>> >        at
>> >
>> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:56)
>> >        at
>> >
>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:379)
>> >        at
>> >
>> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
>> >        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:748)
>> >        at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.checkRegioninfoOnFilesystem(HRegion.java:408)
>> >        at
>> > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:337)
>> >        at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2546)
>> >        at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2532)
>> >        at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2501)
>> >        at
>> > org.apache.hadoop.hbase.util.MetaUtils.openRootRegion(MetaUtils.java:269)
>> >        at
>> > org.apache.hadoop.hbase.util.MetaUtils.scanRootRegion(MetaUtils.java:201)
>> >        at
>> > org.apache.hadoop.hbase.util.Merge.mergeTwoRegions(Merge.java:204)
>> >        at org.apache.hadoop.hbase.util.Merge.run(Merge.java:110)
>> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>> >        at org.apache.hadoop.hbase.util.Merge.main(Merge.java:379)
>> > 2011-09-12 10:21:39,634 INFO  [main.logSyncer] wal.HLog$LogSyncer(973):
>> > main.logSyncer exiting
>> > 2011-09-12 10:21:39,640 ERROR [main] util.Merge(381): exiting due to
>> error
>> > java.lang.IllegalArgumentException: Wrong FS: hdfs://
>> > internalhost.com:54310/hbase/.META./1028785192/.regioninfo, expected:
>> > file:///
>> >        at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:410)
>> >        at
>> >
>> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:56)
>> >        at
>> >
>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:379)
>> >        at
>> >
>> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
>> >        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:748)
>> >        at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.checkRegioninfoOnFilesystem(HRegion.java:408)
>> >        at
>> > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:337)
>> >        at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2546)
>> >        at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2532)
>> >        at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2501)
>> >        at
>> > org.apache.hadoop.hbase.util.MetaUtils.openMetaRegion(MetaUtils.java:276)
>> >        at
>> > org.apache.hadoop.hbase.util.MetaUtils.scanMetaRegion(MetaUtils.java:261)
>> >        at org.apache.hadoop.hbase.util.Merge.run(Merge.java:115)
>> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>> >        at org.apache.hadoop.hbase.util.Merge.main(Merge.java:379)
>> >
>> >
>> > thanks
>> >
>>
>
>
>
> --
>
> - Parmod Mehta
>

Re: HBase region merge problem

Posted by Parmod Mehta <pa...@gmail.com>.
Thanks J-D, the problem was resolved in our dev cluster but have reappeared
in our staging cluster. I added the fs.default.name in the hbase-site.xml as
I did it in the dev cluster with no luck (we running hbase-0.90.4)

    [exec] java.lang.IllegalArgumentException: Wrong FS:
file:/tmp/hbase-root/hbase/-ROOT-/70236052/.regioninfo, expected: hdfs://
staging.server.com:54310
     [exec]     at
org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:410)
     [exec]     at
org.apache.hadoop.hdfs.DistributedFileSystem.checkPath(DistributedFileSystem.java:106)
     [exec]     at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:162)
     [exec]     at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:542)
     [exec]     at
org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:748)
     [exec]     at
org.apache.hadoop.hbase.regionserver.HRegion.checkRegioninfoOnFilesystem(HRegion.java:413)
     [exec]     at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:342)
     [exec]     at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2564)
     [exec]     at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2550)
     [exec]     at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2519)
     [exec]     at
org.apache.hadoop.hbase.util.MetaUtils.openRootRegion(MetaUtils.java:269)
     [exec]     at
org.apache.hadoop.hbase.util.MetaUtils.scanRootRegion(MetaUtils.java:201)
     [exec]     at
org.apache.hadoop.hbase.util.Merge.mergeTwoRegions(Merge.java:204)
     [exec]     at org.apache.hadoop.hbase.util.Merge.run(Merge.java:110)
     [exec]     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
     [exec]     at
com.telcontar.tools.data.util.MergeUtil.runMergeTool(MergeUtil.java:57)
     [exec]     at
com.telcontar.tools.data.util.MergeUtil.run(MergeUtil.java:120)
     [exec]     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
     [exec]     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
     [exec]     at
com.telcontar.tools.data.util.MergeUtil.main(MergeUtil.java:100)
     [exec]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
     [exec]     at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
     [exec]     at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     [exec]     at java.lang.reflect.Method.invoke(Method.java:616)
     [exec]     at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
     [exec] 11/10/17 15:16:33 INFO wal.HLog: main.logSyncer exiting




On Mon, Sep 12, 2011 at 10:45 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:

> Usually it means you need to set fs.default.name in hbase-site.xml
>
> J-D
>
> On Mon, Sep 12, 2011 at 10:33 AM, Parmod Mehta <pa...@gmail.com>
> wrote:
> > Running the merge tool to merge regions on hbase-0.90.1-cdh3u0 run into
> this
> > exception trace. The first INFO level log message I guess is ok (checking
> if
> > the cluster is down).
> >
> > Not sure - why we running into this? bug?
> >
> > java.lang.IllegalArgumentException: Wrong FS: hdfs://
> > internalhost.com:54310/hbase/-ROOT-/70236052/.regioninfo, expected:
> file:///
> >
> >
> > 2011-09-12 10:21:38,364 INFO  [main] util.Merge(82): Verifying that file
> > system is available...
> > 2011-09-12 10:21:38,370 INFO  [main] util.Merge(91): Verifying that HBase
> is
> > not running...
> > 2011-09-12 10:21:38,576 INFO  [main]
> > client.HConnectionManager$HConnectionImplementation(355): ZooKeeper
> > available but no active master location found
> > 2011-09-12 10:21:38,578 INFO  [main]
> > client.HConnectionManager$HConnectionImplementation(372): getMaster
> attempt
> > 0 of 1 failed; no more retrying.
> > org.apache.hadoop.hbase.MasterNotRunningException
> >        at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:356)
> >        at
> > org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:90)
> >        at
> >
> org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:1202)
> >        at org.apache.hadoop.hbase.util.Merge.run(Merge.java:93)
> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> >        at org.apache.hadoop.hbase.util.Merge.main(Merge.java:379)
> > 2011-09-12 10:21:38,913 INFO  [main] util.Merge(199): Merging regions
> > TABLE_2,ERSx0973541,1305737576487.c0a59ee581f619456c962227073f9dd6. and
> > TABLE_2,LEx09136249,1305737576487.85f0adaeb78d362596ea8d178bda5629. in
> table
> > TABLE_2
> > 2011-09-12 10:21:38,940 INFO  [main] wal.HLog(364): HLog configuration:
> > blocksize=32 MB, rollsize=30.4 MB, enabled=true, flushlogentries=1,
> > optionallogflushinternal=1000ms
> > 2011-09-12 10:21:39,016 WARN  [main] util.NativeCodeLoader(52): Unable to
> > load native-hadoop library for your platform... using builtin-java
> classes
> > where applicable
> > 2011-09-12 10:21:39,602 INFO  [main] wal.SequenceFileLogWriter(126):
> Using
> > syncFs -- HDFS-200
> > 2011-09-12 10:21:39,605 INFO  [main] wal.HLog(510): New hlog
> > /home/pmehta/.logs_1315848098931/hlog.1315848098940
> > 2011-09-12 10:21:39,606 INFO  [main] wal.HLog(396):
> > getNumCurrentReplicas--HDFS-826 not available;
> >
> hdfs_out=org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer@3a289d2e
> ,
> >
> exception=org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.getNumCurrentReplicas()
> > 2011-09-12 10:21:39,631 FATAL [main] util.Merge(114): Merge failed
> > java.lang.IllegalArgumentException: Wrong FS: hdfs://
> > internalhost.com:54310/hbase/-ROOT-/70236052/.regioninfo, expected:
> file:///
> >        at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:410)
> >        at
> >
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:56)
> >        at
> >
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:379)
> >        at
> >
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
> >        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:748)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.checkRegioninfoOnFilesystem(HRegion.java:408)
> >        at
> > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:337)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2546)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2532)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2501)
> >        at
> > org.apache.hadoop.hbase.util.MetaUtils.openRootRegion(MetaUtils.java:269)
> >        at
> > org.apache.hadoop.hbase.util.MetaUtils.scanRootRegion(MetaUtils.java:201)
> >        at
> > org.apache.hadoop.hbase.util.Merge.mergeTwoRegions(Merge.java:204)
> >        at org.apache.hadoop.hbase.util.Merge.run(Merge.java:110)
> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> >        at org.apache.hadoop.hbase.util.Merge.main(Merge.java:379)
> > 2011-09-12 10:21:39,634 INFO  [main.logSyncer] wal.HLog$LogSyncer(973):
> > main.logSyncer exiting
> > 2011-09-12 10:21:39,640 ERROR [main] util.Merge(381): exiting due to
> error
> > java.lang.IllegalArgumentException: Wrong FS: hdfs://
> > internalhost.com:54310/hbase/.META./1028785192/.regioninfo, expected:
> > file:///
> >        at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:410)
> >        at
> >
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:56)
> >        at
> >
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:379)
> >        at
> >
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
> >        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:748)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.checkRegioninfoOnFilesystem(HRegion.java:408)
> >        at
> > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:337)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2546)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2532)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2501)
> >        at
> > org.apache.hadoop.hbase.util.MetaUtils.openMetaRegion(MetaUtils.java:276)
> >        at
> > org.apache.hadoop.hbase.util.MetaUtils.scanMetaRegion(MetaUtils.java:261)
> >        at org.apache.hadoop.hbase.util.Merge.run(Merge.java:115)
> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> >        at org.apache.hadoop.hbase.util.Merge.main(Merge.java:379)
> >
> >
> > thanks
> >
>



-- 

- Parmod Mehta

Re: HBase region merge problem

Posted by Jean-Daniel Cryans <jd...@apache.org>.
Usually it means you need to set fs.default.name in hbase-site.xml

J-D

On Mon, Sep 12, 2011 at 10:33 AM, Parmod Mehta <pa...@gmail.com> wrote:
> Running the merge tool to merge regions on hbase-0.90.1-cdh3u0 run into this
> exception trace. The first INFO level log message I guess is ok (checking if
> the cluster is down).
>
> Not sure - why we running into this? bug?
>
> java.lang.IllegalArgumentException: Wrong FS: hdfs://
> internalhost.com:54310/hbase/-ROOT-/70236052/.regioninfo, expected: file:///
>
>
> 2011-09-12 10:21:38,364 INFO  [main] util.Merge(82): Verifying that file
> system is available...
> 2011-09-12 10:21:38,370 INFO  [main] util.Merge(91): Verifying that HBase is
> not running...
> 2011-09-12 10:21:38,576 INFO  [main]
> client.HConnectionManager$HConnectionImplementation(355): ZooKeeper
> available but no active master location found
> 2011-09-12 10:21:38,578 INFO  [main]
> client.HConnectionManager$HConnectionImplementation(372): getMaster attempt
> 0 of 1 failed; no more retrying.
> org.apache.hadoop.hbase.MasterNotRunningException
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:356)
>        at
> org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:90)
>        at
> org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:1202)
>        at org.apache.hadoop.hbase.util.Merge.run(Merge.java:93)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at org.apache.hadoop.hbase.util.Merge.main(Merge.java:379)
> 2011-09-12 10:21:38,913 INFO  [main] util.Merge(199): Merging regions
> TABLE_2,ERSx0973541,1305737576487.c0a59ee581f619456c962227073f9dd6. and
> TABLE_2,LEx09136249,1305737576487.85f0adaeb78d362596ea8d178bda5629. in table
> TABLE_2
> 2011-09-12 10:21:38,940 INFO  [main] wal.HLog(364): HLog configuration:
> blocksize=32 MB, rollsize=30.4 MB, enabled=true, flushlogentries=1,
> optionallogflushinternal=1000ms
> 2011-09-12 10:21:39,016 WARN  [main] util.NativeCodeLoader(52): Unable to
> load native-hadoop library for your platform... using builtin-java classes
> where applicable
> 2011-09-12 10:21:39,602 INFO  [main] wal.SequenceFileLogWriter(126): Using
> syncFs -- HDFS-200
> 2011-09-12 10:21:39,605 INFO  [main] wal.HLog(510): New hlog
> /home/pmehta/.logs_1315848098931/hlog.1315848098940
> 2011-09-12 10:21:39,606 INFO  [main] wal.HLog(396):
> getNumCurrentReplicas--HDFS-826 not available;
> hdfs_out=org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer@3a289d2e,
> exception=org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.getNumCurrentReplicas()
> 2011-09-12 10:21:39,631 FATAL [main] util.Merge(114): Merge failed
> java.lang.IllegalArgumentException: Wrong FS: hdfs://
> internalhost.com:54310/hbase/-ROOT-/70236052/.regioninfo, expected: file:///
>        at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:410)
>        at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:56)
>        at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:379)
>        at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
>        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:748)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.checkRegioninfoOnFilesystem(HRegion.java:408)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:337)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2546)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2532)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2501)
>        at
> org.apache.hadoop.hbase.util.MetaUtils.openRootRegion(MetaUtils.java:269)
>        at
> org.apache.hadoop.hbase.util.MetaUtils.scanRootRegion(MetaUtils.java:201)
>        at
> org.apache.hadoop.hbase.util.Merge.mergeTwoRegions(Merge.java:204)
>        at org.apache.hadoop.hbase.util.Merge.run(Merge.java:110)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at org.apache.hadoop.hbase.util.Merge.main(Merge.java:379)
> 2011-09-12 10:21:39,634 INFO  [main.logSyncer] wal.HLog$LogSyncer(973):
> main.logSyncer exiting
> 2011-09-12 10:21:39,640 ERROR [main] util.Merge(381): exiting due to error
> java.lang.IllegalArgumentException: Wrong FS: hdfs://
> internalhost.com:54310/hbase/.META./1028785192/.regioninfo, expected:
> file:///
>        at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:410)
>        at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:56)
>        at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:379)
>        at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
>        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:748)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.checkRegioninfoOnFilesystem(HRegion.java:408)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:337)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2546)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2532)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2501)
>        at
> org.apache.hadoop.hbase.util.MetaUtils.openMetaRegion(MetaUtils.java:276)
>        at
> org.apache.hadoop.hbase.util.MetaUtils.scanMetaRegion(MetaUtils.java:261)
>        at org.apache.hadoop.hbase.util.Merge.run(Merge.java:115)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at org.apache.hadoop.hbase.util.Merge.main(Merge.java:379)
>
>
> thanks
>