You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by gabriela montiel <ga...@oracle.com> on 2014/08/21 01:18:21 UTC
Create custom filter on HBase 0.96.1.1-cdh5.0.1
Hi all,
I have been working on migrating a custom filter used in HBase 0.94 to
make it work on HBase 0.96.1.1. This custom filter extends the
FilterBase API and receives only two byte arrays. According to the
documentation both toByteArray() and parseFrom(byte[]) should be
implemented. After adding this APIs, copying the jar to all region
servers and restarting HBase, I noticed that the filter is not working
(all records are retrieved). I don't get any errors, but checking out
the trace the constructor and toByteArray() APIs are only called but no
other operation on the filter is executed.
I saw that you need to use FilterProtos from the protobuf to write/read
the byte array to rebuild the filter, however there is no documentation
on how to update these APIs to add my custom filter. Do you have any
idea on what we are doing wrong?
Thanks,
Gaby
Re: Create custom filter on HBase 0.96.1.1-cdh5.0.1
Posted by Ted Yu <yu...@gmail.com>.
As you mentioned before, FilterProtos is used to perform parsing.
You can use the following command to find JIRAs which modified Filter.proto
:
git log hbase-protocol//src/main/protobuf/Filter.proto
One example is:
HBASE-7542 SCVF: Avoid sending two unwanted boolean values from client to RS
BTW please use pastebin - some spaces were missing in the code you pasted.
Cheers
On Wed, Aug 20, 2014 at 6:20 PM, gabriela montiel <
gabriela.montiel@oracle.com> wrote:
>
> Hi all,
>
> Here is the code snippet from the implementation of the parseFrom and
> toByteArray code. Is there something wrong I am doing there?
>
> Thanks,
>
>
> 1.
> publicKeyValueFilter(byte[]key,byte[]value)
> 2.
> {
> 3.
> m_keyBytes=key;
> 4.
> m_valueBytes=value;
> 5.
> }
> 6.
> 7.
> 8.
> /**
> 9.
> * Filter the row if the K/V pair is not found in the row data.
> 10.
> *
> 11.
> * @param cell the cell in question
> 12.
> *
> 13.
> */
> 14.
> publicFilter.ReturnCodefilterKeyValue(Cell cell)
> 15.
> {
> 16.
> if(Bytes.equals(m_keyBytes, cell.getQualifier())){//
> CellUtil.cloneQualifier(cell))) {
> 17.
> 18.
> byte[]valueBytes=cell.getValue();// CellUtil.cloneValue(cell);
> 19.
> // found the matched k/v, do not filter this row
> 20.
> if(Bytes.equals(m_valueBytes, valueBytes)){
> 21.
> debug("filterKeyValue: row matches K/V pair");
> 22.
> 23.
> m_bFilterRow=false;
> 24.
> }
> 25.
> }
> 26.
> else{
> 27.
> debug("filterKeyValue: row not matches key");
> 28.
> }
> 29.
> 30.
> // include all columns
> 31.
> returnFilter.ReturnCode.INCLUDE;
> 32.
> }
> 33.
> 34.
> publicstatic Filter
> parseFrom(byte[]pbBytes)throwsDeserializationException
> 35.
> {
> 36.
> try{
> 37.
> ByteArrayInputStreambais=newByteArrayInputStream(pbBytes);
> 38.
> DataInputStreaminput=newDataInputStream(bais);
> 39.
> 40.
> debug("parseFrom: read key byte array from byte array");
> 41.
> byte[]keyBytes=Bytes.readByteArray(input);
> 42.
> 43.
> debug("parseFrom: read value byte array from byte array");
> 44.
> byte[]valueBytes=Bytes.readByteArray(input);
> 45.
> 46.
> returnnewKeyValueFilter(keyBytes, valueBytes);
> 47.
> }
> 48.
> catch(IOExceptione){
> 49.
> thrownewDeserializationException(e);
> 50.
> }
> 51.
> }
> 52.
> 53.
> publicbyte[]toByteArray()throwsIOException
> 54.
> {
> 55.
> ByteArrayOutputStreambaos=newByteArrayOutputStream(60/*size*/);
> 56.
> DataOutputStreamdos=newDataOutputStream(baos);
> 57.
> 58.
> debug("toByteArray: write key into byte array stream");
> 59.
> Bytes.writeByteArray(dos, m_keyBytes);
> 60.
> 61.
> debug("toByteArray: write value into byte array stream");
> 62.
> Bytes.writeByteArray(dos, m_valueBytes);
> 63.
> dos.flush();
> 64.
> baos.flush();
> 65.
> 66.
> returnbaos.toByteArray();
> 67.
>
> }
>
>
>
>
> On 20/08/2014 06:46 p.m., Ted Yu wrote:
>
>> Can you show implementation for parseFrom(byte[]) - using pastebin ?
>>
>> If possible, seeing the code for whole class would help us understand
>> better.
>>
>> Cheers
>>
>>
>> On Wed, Aug 20, 2014 at 4:18 PM, gabriela montiel <
>> gabriela.montiel@oracle.com> wrote:
>>
>> Hi all,
>>>
>>> I have been working on migrating a custom filter used in HBase 0.94 to
>>> make it work on HBase 0.96.1.1. This custom filter extends the FilterBase
>>> API and receives only two byte arrays. According to the documentation
>>> both
>>> toByteArray() and parseFrom(byte[]) should be implemented. After adding
>>> this APIs, copying the jar to all region servers and restarting HBase, I
>>> noticed that the filter is not working (all records are retrieved). I
>>> don't
>>> get any errors, but checking out the trace the constructor and
>>> toByteArray() APIs are only called but no other operation on the filter
>>> is
>>> executed.
>>>
>>> I saw that you need to use FilterProtos from the protobuf to write/read
>>> the byte array to rebuild the filter, however there is no documentation
>>> on
>>> how to update these APIs to add my custom filter. Do you have any idea on
>>> what we are doing wrong?
>>>
>>> Thanks,
>>>
>>> Gaby
>>>
>>>
>
Re: Create custom filter on HBase 0.96.1.1-cdh5.0.1
Posted by gabriela montiel <ga...@oracle.com>.
Hi all,
Here is the code snippet from the implementation of the parseFrom and
toByteArray code. Is there something wrong I am doing there?
Thanks,
1.
publicKeyValueFilter(byte[]key,byte[]value)
2.
{
3.
m_keyBytes=key;
4.
m_valueBytes=value;
5.
}
6.
7.
8.
/**
9.
* Filter the row if the K/V pair is not found in the row data.
10.
*
11.
* @param cell the cell in question
12.
*
13.
*/
14.
publicFilter.ReturnCodefilterKeyValue(Cell cell)
15.
{
16.
if(Bytes.equals(m_keyBytes, cell.getQualifier())){//
CellUtil.cloneQualifier(cell))) {
17.
18.
byte[]valueBytes=cell.getValue();// CellUtil.cloneValue(cell);
19.
// found the matched k/v, do not filter this row
20.
if(Bytes.equals(m_valueBytes, valueBytes)){
21.
debug("filterKeyValue: row matches K/V pair");
22.
23.
m_bFilterRow=false;
24.
}
25.
}
26.
else{
27.
debug("filterKeyValue: row not matches key");
28.
}
29.
30.
// include all columns
31.
returnFilter.ReturnCode.INCLUDE;
32.
}
33.
34.
publicstatic Filter
parseFrom(byte[]pbBytes)throwsDeserializationException
35.
{
36.
try{
37.
ByteArrayInputStreambais=newByteArrayInputStream(pbBytes);
38.
DataInputStreaminput=newDataInputStream(bais);
39.
40.
debug("parseFrom: read key byte array from byte array");
41.
byte[]keyBytes=Bytes.readByteArray(input);
42.
43.
debug("parseFrom: read value byte array from byte array");
44.
byte[]valueBytes=Bytes.readByteArray(input);
45.
46.
returnnewKeyValueFilter(keyBytes, valueBytes);
47.
}
48.
catch(IOExceptione){
49.
thrownewDeserializationException(e);
50.
}
51.
}
52.
53.
publicbyte[]toByteArray()throwsIOException
54.
{
55.
ByteArrayOutputStreambaos=newByteArrayOutputStream(60/*size*/);
56.
DataOutputStreamdos=newDataOutputStream(baos);
57.
58.
debug("toByteArray: write key into byte array stream");
59.
Bytes.writeByteArray(dos, m_keyBytes);
60.
61.
debug("toByteArray: write value into byte array stream");
62.
Bytes.writeByteArray(dos, m_valueBytes);
63.
dos.flush();
64.
baos.flush();
65.
66.
returnbaos.toByteArray();
67.
}
On 20/08/2014 06:46 p.m., Ted Yu wrote:
> Can you show implementation for parseFrom(byte[]) - using pastebin ?
>
> If possible, seeing the code for whole class would help us understand
> better.
>
> Cheers
>
>
> On Wed, Aug 20, 2014 at 4:18 PM, gabriela montiel <
> gabriela.montiel@oracle.com> wrote:
>
>> Hi all,
>>
>> I have been working on migrating a custom filter used in HBase 0.94 to
>> make it work on HBase 0.96.1.1. This custom filter extends the FilterBase
>> API and receives only two byte arrays. According to the documentation both
>> toByteArray() and parseFrom(byte[]) should be implemented. After adding
>> this APIs, copying the jar to all region servers and restarting HBase, I
>> noticed that the filter is not working (all records are retrieved). I don't
>> get any errors, but checking out the trace the constructor and
>> toByteArray() APIs are only called but no other operation on the filter is
>> executed.
>>
>> I saw that you need to use FilterProtos from the protobuf to write/read
>> the byte array to rebuild the filter, however there is no documentation on
>> how to update these APIs to add my custom filter. Do you have any idea on
>> what we are doing wrong?
>>
>> Thanks,
>>
>> Gaby
>>
Re: Create custom filter on HBase 0.96.1.1-cdh5.0.1
Posted by Ted Yu <yu...@gmail.com>.
Can you show implementation for parseFrom(byte[]) - using pastebin ?
If possible, seeing the code for whole class would help us understand
better.
Cheers
On Wed, Aug 20, 2014 at 4:18 PM, gabriela montiel <
gabriela.montiel@oracle.com> wrote:
> Hi all,
>
> I have been working on migrating a custom filter used in HBase 0.94 to
> make it work on HBase 0.96.1.1. This custom filter extends the FilterBase
> API and receives only two byte arrays. According to the documentation both
> toByteArray() and parseFrom(byte[]) should be implemented. After adding
> this APIs, copying the jar to all region servers and restarting HBase, I
> noticed that the filter is not working (all records are retrieved). I don't
> get any errors, but checking out the trace the constructor and
> toByteArray() APIs are only called but no other operation on the filter is
> executed.
>
> I saw that you need to use FilterProtos from the protobuf to write/read
> the byte array to rebuild the filter, however there is no documentation on
> how to update these APIs to add my custom filter. Do you have any idea on
> what we are doing wrong?
>
> Thanks,
>
> Gaby
>
Re: Create custom filter on HBase 0.96.1.1-cdh5.0.1
Posted by Ted Yu <yu...@gmail.com>.
Can you show implementation for parseFrom(byte[]) - using pastebin ?
If possible, seeing the code for whole class would help us understand
better.
Cheers
On Wed, Aug 20, 2014 at 4:18 PM, gabriela montiel <
gabriela.montiel@oracle.com> wrote:
> Hi all,
>
> I have been working on migrating a custom filter used in HBase 0.94 to
> make it work on HBase 0.96.1.1. This custom filter extends the FilterBase
> API and receives only two byte arrays. According to the documentation both
> toByteArray() and parseFrom(byte[]) should be implemented. After adding
> this APIs, copying the jar to all region servers and restarting HBase, I
> noticed that the filter is not working (all records are retrieved). I don't
> get any errors, but checking out the trace the constructor and
> toByteArray() APIs are only called but no other operation on the filter is
> executed.
>
> I saw that you need to use FilterProtos from the protobuf to write/read
> the byte array to rebuild the filter, however there is no documentation on
> how to update these APIs to add my custom filter. Do you have any idea on
> what we are doing wrong?
>
> Thanks,
>
> Gaby
>