You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Ted Yu <yu...@gmail.com> on 2011/04/19 23:05:48 UTC
using RegionSplitter
Hi,
In 0.90.2, I tried to use RegionSplitter:
[hadoop@us01-ciqps1-name01 hbase]$ bin/hbase
org.apache.hadoop.hbase.util.RegionSplitter -r -o 2
'M2M-INTEGRATION-DIMENSIONS-1303180666422'
...
11/04/19 21:02:33 DEBUG client.HConnectionManager$HConnectionImplementation:
Cache hit for row <> in tableName M2M-INTEGRATION-DIMENSIONS-1303180666422:
location server us01-ciqps1-grid07.carrieriq.com:60020, location region name
M2M-INTEGRATION-DIMENSIONS-1303180666422,,1303180669457.19be2774054a672e905554bcaf73d5e0.
Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS:
hdfs://
us01-ciqps1-name01.carrieriq.com:9000/hbase/M2M-INTEGRATION-DIMENSIONS-1303180666422/_balancedSplit,
expected: file:///
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:352)
at
org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:47)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:368)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:690)
at
org.apache.hadoop.hbase.util.RegionSplitter.getSplits(RegionSplitter.java:612)
at
org.apache.hadoop.hbase.util.RegionSplitter.rollingSplit(RegionSplitter.java:362)
at
org.apache.hadoop.hbase.util.RegionSplitter.main(RegionSplitter.java:291)
Has anyone experienced this ?
Thanks
Re: using RegionSplitter
Posted by Ted Yu <yu...@gmail.com>.
There is assumption that start and end keys can be converted to BigInteger:
public byte[] split(byte[] start, byte[] end) {
BigInteger s = convertToBigInteger(start);
BigInteger e = convertToBigInteger(end);
The assumption doesn't hold in our use case.
On Tue, Apr 19, 2011 at 2:24 PM, Ted Yu <yu...@gmail.com> wrote:
> Thanks J-D.
>
> Now I see:
> 11/04/19 21:22:03 DEBUG util.RegionSplitter: No _balancedSplit file.
> Calculating splits...
> 11/04/19 21:22:03 DEBUG client.MetaScanner: Scanning .META. starting at
> row=M2M-INTEGRATION-DIMENSIONS-1303180666422,,00000000000000 for
> max=2147483647 rows
> 11/04/19 21:22:04 DEBUG util.RegionSplitter: Table
> M2M-INTEGRATION-DIMENSIONS-1303180666422 has 5 regions that will be split.
> Exception in thread "main" java.lang.NumberFormatException: For input
> string: ">"
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
> at java.lang.Integer.parseInt(Integer.java:449)
> at java.math.BigInteger.<init>(BigInteger.java:316)
> at
> org.apache.hadoop.hbase.util.RegionSplitter$MD5StringSplit.convertToBigInteger(RegionSplitter.java:779)
> at
> org.apache.hadoop.hbase.util.RegionSplitter$MD5StringSplit.split(RegionSplitter.java:704)
> at
> org.apache.hadoop.hbase.util.RegionSplitter.getSplits(RegionSplitter.java:639)
>
> at
> org.apache.hadoop.hbase.util.RegionSplitter.rollingSplit(RegionSplitter.java:362)
> at
> org.apache.hadoop.hbase.util.RegionSplitter.main(RegionSplitter.java:291)
>
> On Tue, Apr 19, 2011 at 2:12 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> In general IIRC that message means that you need to have your hadoop
>> configuration folder in HBase's classpath.
>>
>> J-D
>>
>> On Tue, Apr 19, 2011 at 2:05 PM, Ted Yu <yu...@gmail.com> wrote:
>> > Hi,
>> > In 0.90.2, I tried to use RegionSplitter:
>> > [hadoop@us01-ciqps1-name01 hbase]$ bin/hbase
>> > org.apache.hadoop.hbase.util.RegionSplitter -r -o 2
>> > 'M2M-INTEGRATION-DIMENSIONS-1303180666422'
>> > ...
>> > 11/04/19 21:02:33 DEBUG
>> client.HConnectionManager$HConnectionImplementation:
>> > Cache hit for row <> in tableName
>> M2M-INTEGRATION-DIMENSIONS-1303180666422:
>> > location server us01-ciqps1-grid07.carrieriq.com:60020, location region
>> name
>> >
>> M2M-INTEGRATION-DIMENSIONS-1303180666422,,1303180669457.19be2774054a672e905554bcaf73d5e0.
>> > Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS:
>> > hdfs://
>> >
>> us01-ciqps1-name01.carrieriq.com:9000/hbase/M2M-INTEGRATION-DIMENSIONS-1303180666422/_balancedSplit
>> ,
>> > expected: file:///
>> > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:352)
>> > at
>> >
>> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:47)
>> > at
>> >
>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:368)
>> > at
>> >
>> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
>> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:690)
>> > at
>> >
>> org.apache.hadoop.hbase.util.RegionSplitter.getSplits(RegionSplitter.java:612)
>> > at
>> >
>> org.apache.hadoop.hbase.util.RegionSplitter.rollingSplit(RegionSplitter.java:362)
>> > at
>> >
>> org.apache.hadoop.hbase.util.RegionSplitter.main(RegionSplitter.java:291)
>> >
>> > Has anyone experienced this ?
>> >
>> > Thanks
>> >
>>
>
>
Re: using RegionSplitter
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Nothing comes to mind, I'd have to read the code.
J-D
On Tue, Apr 19, 2011 at 2:24 PM, Ted Yu <yu...@gmail.com> wrote:
> Thanks J-D.
>
> Now I see:
> 11/04/19 21:22:03 DEBUG util.RegionSplitter: No _balancedSplit file.
> Calculating splits...
> 11/04/19 21:22:03 DEBUG client.MetaScanner: Scanning .META. starting at
> row=M2M-INTEGRATION-DIMENSIONS-1303180666422,,00000000000000 for
> max=2147483647 rows
> 11/04/19 21:22:04 DEBUG util.RegionSplitter: Table
> M2M-INTEGRATION-DIMENSIONS-1303180666422 has 5 regions that will be split.
> Exception in thread "main" java.lang.NumberFormatException: For input
> string: ">"
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
> at java.lang.Integer.parseInt(Integer.java:449)
> at java.math.BigInteger.<init>(BigInteger.java:316)
> at
> org.apache.hadoop.hbase.util.RegionSplitter$MD5StringSplit.convertToBigInteger(RegionSplitter.java:779)
> at
> org.apache.hadoop.hbase.util.RegionSplitter$MD5StringSplit.split(RegionSplitter.java:704)
> at
> org.apache.hadoop.hbase.util.RegionSplitter.getSplits(RegionSplitter.java:639)
> at
> org.apache.hadoop.hbase.util.RegionSplitter.rollingSplit(RegionSplitter.java:362)
> at
> org.apache.hadoop.hbase.util.RegionSplitter.main(RegionSplitter.java:291)
>
> On Tue, Apr 19, 2011 at 2:12 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> In general IIRC that message means that you need to have your hadoop
>> configuration folder in HBase's classpath.
>>
>> J-D
>>
>> On Tue, Apr 19, 2011 at 2:05 PM, Ted Yu <yu...@gmail.com> wrote:
>> > Hi,
>> > In 0.90.2, I tried to use RegionSplitter:
>> > [hadoop@us01-ciqps1-name01 hbase]$ bin/hbase
>> > org.apache.hadoop.hbase.util.RegionSplitter -r -o 2
>> > 'M2M-INTEGRATION-DIMENSIONS-1303180666422'
>> > ...
>> > 11/04/19 21:02:33 DEBUG
>> client.HConnectionManager$HConnectionImplementation:
>> > Cache hit for row <> in tableName
>> M2M-INTEGRATION-DIMENSIONS-1303180666422:
>> > location server us01-ciqps1-grid07.carrieriq.com:60020, location region
>> name
>> >
>> M2M-INTEGRATION-DIMENSIONS-1303180666422,,1303180669457.19be2774054a672e905554bcaf73d5e0.
>> > Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS:
>> > hdfs://
>> >
>> us01-ciqps1-name01.carrieriq.com:9000/hbase/M2M-INTEGRATION-DIMENSIONS-1303180666422/_balancedSplit
>> ,
>> > expected: file:///
>> > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:352)
>> > at
>> >
>> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:47)
>> > at
>> >
>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:368)
>> > at
>> >
>> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
>> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:690)
>> > at
>> >
>> org.apache.hadoop.hbase.util.RegionSplitter.getSplits(RegionSplitter.java:612)
>> > at
>> >
>> org.apache.hadoop.hbase.util.RegionSplitter.rollingSplit(RegionSplitter.java:362)
>> > at
>> > org.apache.hadoop.hbase.util.RegionSplitter.main(RegionSplitter.java:291)
>> >
>> > Has anyone experienced this ?
>> >
>> > Thanks
>> >
>>
>
Re: using RegionSplitter
Posted by Ted Yu <yu...@gmail.com>.
Thanks J-D.
Now I see:
11/04/19 21:22:03 DEBUG util.RegionSplitter: No _balancedSplit file.
Calculating splits...
11/04/19 21:22:03 DEBUG client.MetaScanner: Scanning .META. starting at
row=M2M-INTEGRATION-DIMENSIONS-1303180666422,,00000000000000 for
max=2147483647 rows
11/04/19 21:22:04 DEBUG util.RegionSplitter: Table
M2M-INTEGRATION-DIMENSIONS-1303180666422 has 5 regions that will be split.
Exception in thread "main" java.lang.NumberFormatException: For input
string: ">"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
at java.lang.Integer.parseInt(Integer.java:449)
at java.math.BigInteger.<init>(BigInteger.java:316)
at
org.apache.hadoop.hbase.util.RegionSplitter$MD5StringSplit.convertToBigInteger(RegionSplitter.java:779)
at
org.apache.hadoop.hbase.util.RegionSplitter$MD5StringSplit.split(RegionSplitter.java:704)
at
org.apache.hadoop.hbase.util.RegionSplitter.getSplits(RegionSplitter.java:639)
at
org.apache.hadoop.hbase.util.RegionSplitter.rollingSplit(RegionSplitter.java:362)
at
org.apache.hadoop.hbase.util.RegionSplitter.main(RegionSplitter.java:291)
On Tue, Apr 19, 2011 at 2:12 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> In general IIRC that message means that you need to have your hadoop
> configuration folder in HBase's classpath.
>
> J-D
>
> On Tue, Apr 19, 2011 at 2:05 PM, Ted Yu <yu...@gmail.com> wrote:
> > Hi,
> > In 0.90.2, I tried to use RegionSplitter:
> > [hadoop@us01-ciqps1-name01 hbase]$ bin/hbase
> > org.apache.hadoop.hbase.util.RegionSplitter -r -o 2
> > 'M2M-INTEGRATION-DIMENSIONS-1303180666422'
> > ...
> > 11/04/19 21:02:33 DEBUG
> client.HConnectionManager$HConnectionImplementation:
> > Cache hit for row <> in tableName
> M2M-INTEGRATION-DIMENSIONS-1303180666422:
> > location server us01-ciqps1-grid07.carrieriq.com:60020, location region
> name
> >
> M2M-INTEGRATION-DIMENSIONS-1303180666422,,1303180669457.19be2774054a672e905554bcaf73d5e0.
> > Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS:
> > hdfs://
> >
> us01-ciqps1-name01.carrieriq.com:9000/hbase/M2M-INTEGRATION-DIMENSIONS-1303180666422/_balancedSplit
> ,
> > expected: file:///
> > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:352)
> > at
> >
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:47)
> > at
> >
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:368)
> > at
> >
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:690)
> > at
> >
> org.apache.hadoop.hbase.util.RegionSplitter.getSplits(RegionSplitter.java:612)
> > at
> >
> org.apache.hadoop.hbase.util.RegionSplitter.rollingSplit(RegionSplitter.java:362)
> > at
> > org.apache.hadoop.hbase.util.RegionSplitter.main(RegionSplitter.java:291)
> >
> > Has anyone experienced this ?
> >
> > Thanks
> >
>
Re: using RegionSplitter
Posted by Jean-Daniel Cryans <jd...@apache.org>.
In general IIRC that message means that you need to have your hadoop
configuration folder in HBase's classpath.
J-D
On Tue, Apr 19, 2011 at 2:05 PM, Ted Yu <yu...@gmail.com> wrote:
> Hi,
> In 0.90.2, I tried to use RegionSplitter:
> [hadoop@us01-ciqps1-name01 hbase]$ bin/hbase
> org.apache.hadoop.hbase.util.RegionSplitter -r -o 2
> 'M2M-INTEGRATION-DIMENSIONS-1303180666422'
> ...
> 11/04/19 21:02:33 DEBUG client.HConnectionManager$HConnectionImplementation:
> Cache hit for row <> in tableName M2M-INTEGRATION-DIMENSIONS-1303180666422:
> location server us01-ciqps1-grid07.carrieriq.com:60020, location region name
> M2M-INTEGRATION-DIMENSIONS-1303180666422,,1303180669457.19be2774054a672e905554bcaf73d5e0.
> Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS:
> hdfs://
> us01-ciqps1-name01.carrieriq.com:9000/hbase/M2M-INTEGRATION-DIMENSIONS-1303180666422/_balancedSplit,
> expected: file:///
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:352)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:47)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:368)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:690)
> at
> org.apache.hadoop.hbase.util.RegionSplitter.getSplits(RegionSplitter.java:612)
> at
> org.apache.hadoop.hbase.util.RegionSplitter.rollingSplit(RegionSplitter.java:362)
> at
> org.apache.hadoop.hbase.util.RegionSplitter.main(RegionSplitter.java:291)
>
> Has anyone experienced this ?
>
> Thanks
>