You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Nageswaran Surendhar <na...@gmail.com> on 2011/12/30 09:56:39 UTC
HBase Custom Comparator
public class CustomComparator extends WritableByteArrayComparable
{
final CimKeyType cimKeyType;
final byte[] value;
public CustomComparator(byte[] value, CimKeyType cimKeyType)
{
super(value);
this.value = value;
this.cimKeyType = cimKeyType;
}
@Override
public int compareTo(byte[] value)
{
int differenceValue;
switch (cimKeyType)
{
case INTEGER:
differenceValue = Bytes.toInt(this.value) - Bytes.toInt(value);
break;
case LONG:
differenceValue = (int) (Bytes.toLong(this.value) -
Bytes.toLong(value));
break;
case DOUBLE:
differenceValue = (int) (Bytes.toDouble(this.value) -
Bytes.toDouble(value));
break;
case STRING:
default:
differenceValue =
Bytes.toString(this.value).compareTo(Bytes.toString(value));
}
System.out.println("differenceValue : " +differenceValue);
return differenceValue;
}
}
CustomComparator comparator = new CustomComparator(filterValue,
lField.getCimKeyType());
//BinaryComparator comparator = new BinaryComparator(filterValue);
SingleColumnValueFilter singleColumnValueFilter = new
SingleColumnValueFilter(Bytes.toBytes(familyAndCol[0]),
Bytes.toBytes(familyAndCol[1]), compareOperator, comparator);
if (myNumberOfVersions > 0)
get.setMaxVersions(myNumberOfVersions);
if (filterList != null)
get.setFilter(filterList);
Result result = hTable.get(get);
I m using the above code inside hadoop-mapper. I had make it jar and placed
it inside both hadoop/lib and hbase/lib.
If i use BinaryComparator, i m not getting any exception, and error. But if
I used CustomComparator, I m getting the follwoing Exception.
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException:
Trying to contact region server localhost:50682 for region
mytable,,1325226129670.708fa93842d13d3448295d88906d7ab8., row '444',
but failed after 10 attempts.
Exceptions:
java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
local exception: java.io.EOFException
java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
local exception: java.io.EOFException
java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
local exception: java.io.EOFException
java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
local exception: java.io.EOFException
java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
local exception: java.io.EOFException
java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
local exception: java.io.EOFException
java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
local exception: java.io.EOFException
java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
local exception: java.io.EOFException
java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
local exception: java.io.EOFException
java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
local exception: java.io.EOFException
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1024)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:555)
at
... 12 more
2011-12-30 13:36:30,175 INFO org.apache.hadoop.mapred.Task:
Runnning cleanup for the task
command line jps output is
18667 HMaster
17929 NameNode
21942 Jps
18266 SecondaryNameNode
18100 DataNode
19143 Main
18523 TaskTracker
18362 JobTracker
8641
here mytable is my hbase table name and 444 is rowkey
Development environment : Ubuntu-11.10, Java-6, Hadoop and Hbase from
Cloudera CDH3U2
--
Regards,
Nageswaran S
Re: HBase Custom Comparator
Posted by Nageswaran Surendhar <na...@gmail.com>.
Hi,
I have identified the problem, and fixed it. The problem is, I have to
provide default constructor for CustomComparator...
Regards,
Nageswaran. S
On Wed, Jan 4, 2012 at 3:54 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> Is your comparator on the region server's classpath? Else it won't be
> able to guess what that comparator looks like :)
>
> Also your region server log should tell you it's not able to find
> CustomComparator.
>
> J-D
>
> On Fri, Dec 30, 2011 at 12:56 AM, Nageswaran Surendhar
> <na...@gmail.com> wrote:
> > public class CustomComparator extends WritableByteArrayComparable
> > {
> >
> > final CimKeyType cimKeyType;
> > final byte[] value;
> >
> > public CustomComparator(byte[] value, CimKeyType cimKeyType)
> > {
> > super(value);
> > this.value = value;
> > this.cimKeyType = cimKeyType;
> > }
> >
> > @Override
> > public int compareTo(byte[] value)
> > {
> > int differenceValue;
> >
> > switch (cimKeyType)
> > {
> > case INTEGER:
> > differenceValue = Bytes.toInt(this.value) -
> Bytes.toInt(value);
> > break;
> > case LONG:
> > differenceValue = (int) (Bytes.toLong(this.value) -
> > Bytes.toLong(value));
> > break;
> > case DOUBLE:
> > differenceValue = (int) (Bytes.toDouble(this.value) -
> > Bytes.toDouble(value));
> > break;
> > case STRING:
> > default:
> > differenceValue =
> > Bytes.toString(this.value).compareTo(Bytes.toString(value));
> > }
> >
> > System.out.println("differenceValue : " +differenceValue);
> >
> > return differenceValue;
> > }
> > }
> >
> >
> >
> > CustomComparator comparator = new CustomComparator(filterValue,
> > lField.getCimKeyType());
> > //BinaryComparator comparator = new BinaryComparator(filterValue);
> >
> > SingleColumnValueFilter singleColumnValueFilter = new
> > SingleColumnValueFilter(Bytes.toBytes(familyAndCol[0]),
> > Bytes.toBytes(familyAndCol[1]), compareOperator, comparator);
> >
> > if (myNumberOfVersions > 0)
> > get.setMaxVersions(myNumberOfVersions);
> >
> >
> > if (filterList != null)
> > get.setFilter(filterList);
> >
> > Result result = hTable.get(get);
> >
> >
> > I m using the above code inside hadoop-mapper. I had make it jar and
> placed
> > it inside both hadoop/lib and hbase/lib.
> >
> > If i use BinaryComparator, i m not getting any exception, and error. But
> if
> > I used CustomComparator, I m getting the follwoing Exception.
> >
> >
> > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
> > at org.apache.hadoop.mapred.Child.main(Child.java:264)
> > Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException:
> > Trying to contact region server localhost:50682 for region
> > mytable,,1325226129670.708fa93842d13d3448295d88906d7ab8., row '444',
> > but failed after 10 attempts.
> > Exceptions:
> > java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
> > local exception: java.io.EOFException
> > java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
> > local exception: java.io.EOFException
> > java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
> > local exception: java.io.EOFException
> > java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
> > local exception: java.io.EOFException
> > java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
> > local exception: java.io.EOFException
> > java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
> > local exception: java.io.EOFException
> > java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
> > local exception: java.io.EOFException
> > java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
> > local exception: java.io.EOFException
> > java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
> > local exception: java.io.EOFException
> > java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
> > local exception: java.io.EOFException
> >
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1024)
> > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:555)
> > at
> > ... 12 more
> > 2011-12-30 13:36:30,175 INFO org.apache.hadoop.mapred.Task:
> > Runnning cleanup for the task
> >
> > command line jps output is
> > 18667 HMaster
> > 17929 NameNode
> > 21942 Jps
> > 18266 SecondaryNameNode
> > 18100 DataNode
> > 19143 Main
> > 18523 TaskTracker
> > 18362 JobTracker
> > 8641
> >
> >
> > here mytable is my hbase table name and 444 is rowkey
> >
> > Development environment : Ubuntu-11.10, Java-6, Hadoop and Hbase from
> > Cloudera CDH3U2
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > --
> >
> > Regards,
> > Nageswaran S
>
--
Regards,
Nageswaran S
Ericsson
+91 95669 13981
Re: HBase Custom Comparator
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Is your comparator on the region server's classpath? Else it won't be
able to guess what that comparator looks like :)
Also your region server log should tell you it's not able to find
CustomComparator.
J-D
On Fri, Dec 30, 2011 at 12:56 AM, Nageswaran Surendhar
<na...@gmail.com> wrote:
> public class CustomComparator extends WritableByteArrayComparable
> {
>
> final CimKeyType cimKeyType;
> final byte[] value;
>
> public CustomComparator(byte[] value, CimKeyType cimKeyType)
> {
> super(value);
> this.value = value;
> this.cimKeyType = cimKeyType;
> }
>
> @Override
> public int compareTo(byte[] value)
> {
> int differenceValue;
>
> switch (cimKeyType)
> {
> case INTEGER:
> differenceValue = Bytes.toInt(this.value) - Bytes.toInt(value);
> break;
> case LONG:
> differenceValue = (int) (Bytes.toLong(this.value) -
> Bytes.toLong(value));
> break;
> case DOUBLE:
> differenceValue = (int) (Bytes.toDouble(this.value) -
> Bytes.toDouble(value));
> break;
> case STRING:
> default:
> differenceValue =
> Bytes.toString(this.value).compareTo(Bytes.toString(value));
> }
>
> System.out.println("differenceValue : " +differenceValue);
>
> return differenceValue;
> }
> }
>
>
>
> CustomComparator comparator = new CustomComparator(filterValue,
> lField.getCimKeyType());
> //BinaryComparator comparator = new BinaryComparator(filterValue);
>
> SingleColumnValueFilter singleColumnValueFilter = new
> SingleColumnValueFilter(Bytes.toBytes(familyAndCol[0]),
> Bytes.toBytes(familyAndCol[1]), compareOperator, comparator);
>
> if (myNumberOfVersions > 0)
> get.setMaxVersions(myNumberOfVersions);
>
>
> if (filterList != null)
> get.setFilter(filterList);
>
> Result result = hTable.get(get);
>
>
> I m using the above code inside hadoop-mapper. I had make it jar and placed
> it inside both hadoop/lib and hbase/lib.
>
> If i use BinaryComparator, i m not getting any exception, and error. But if
> I used CustomComparator, I m getting the follwoing Exception.
>
>
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
> at org.apache.hadoop.mapred.Child.main(Child.java:264)
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException:
> Trying to contact region server localhost:50682 for region
> mytable,,1325226129670.708fa93842d13d3448295d88906d7ab8., row '444',
> but failed after 10 attempts.
> Exceptions:
> java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
> local exception: java.io.EOFException
> java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
> local exception: java.io.EOFException
> java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
> local exception: java.io.EOFException
> java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
> local exception: java.io.EOFException
> java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
> local exception: java.io.EOFException
> java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
> local exception: java.io.EOFException
> java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
> local exception: java.io.EOFException
> java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
> local exception: java.io.EOFException
> java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
> local exception: java.io.EOFException
> java.io.IOException: Call to localhost/127.0.0.1:50682 failed on
> local exception: java.io.EOFException
>
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1024)
> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:555)
> at
> ... 12 more
> 2011-12-30 13:36:30,175 INFO org.apache.hadoop.mapred.Task:
> Runnning cleanup for the task
>
> command line jps output is
> 18667 HMaster
> 17929 NameNode
> 21942 Jps
> 18266 SecondaryNameNode
> 18100 DataNode
> 19143 Main
> 18523 TaskTracker
> 18362 JobTracker
> 8641
>
>
> here mytable is my hbase table name and 444 is rowkey
>
> Development environment : Ubuntu-11.10, Java-6, Hadoop and Hbase from
> Cloudera CDH3U2
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> Regards,
> Nageswaran S