You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Matt K <ma...@gmail.com> on 2014/10/15 23:08:08 UTC
custom filter on hbase 0.96
Hi all,
I'm trying to get a custom filter to work on HBase 0.96. After some
searching, I found that starting from 0.96, the implementer is required to
implement "toByteArray" and "parseFrom" methods, using Protocol Buffers.
But I'm having trouble with the "how".
The "proto" file for the existing filters is located here:
https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/protobuf/Filter.proto
Am I supposed to modify that file? Or create a new file, compile it into
Java, and package it up with the filter?
In the meantime, I've taken a shortcut that's not working. Here's my code:
http://pastebin.com/iHFKu9Xz
I'm using "PrefixFilter", which comes with HBase, since I'm also filtering
by "prefix". However, that errors out with the following:
http://pastebin.com/zBg47p6Z
Thanks in advance for helping!
-Matt
Re: custom filter on hbase 0.96
Posted by Ted Yu <yu...@gmail.com>.
See this blog post:
http://www.flurry.com/2012/12/06/exploring-dynamic-loading-of-custom-filters-i#.VEcNtNR4rZg
Cheers
On Tue, Oct 21, 2014 at 6:48 PM, Kevin <ke...@gmail.com> wrote:
> Also, if you do end up using dynamic loading, you'll need a way to version
> your filters because the RS will not reload a JAR if it changes.
>
> On Tue, Oct 21, 2014 at 9:46 PM, Kevin <ke...@gmail.com> wrote:
>
> > I haven't tried dynamic loading of filters on RS, but I know it does
> > exist. See https://issues.apache.org/jira/browse/HBASE-9301.
> >
> > If you still can't get it to work, then I suggest distributing your
> > filters to the RS and restart them. Let us know how everything works out.
> >
> > On Tue, Oct 21, 2014 at 9:02 PM, Matt K <ma...@gmail.com> wrote:
> >
> >> Thanks Kevin!
> >>
> >> I was under impression, probably mistakingly, that as of 0.96 placing
> >> the filter on hdfs under hbase lib directory is sufficient and RS should
> >> load the filter dynamically from hdfs. Is that not the case?
> >>
> >> On Tuesday, October 21, 2014, Kevin <ke...@gmail.com> wrote:
> >>
> >> > BTW, the error looks like you didn't distribute your custom filter to
> >> your
> >> > region servers.
> >> >
> >> > On Tue, Oct 21, 2014 at 1:34 PM, Kevin <kevin.macksamie@gmail.com
> >> > <javascript:;>> wrote:
> >> >
> >> > > Matt,
> >> > >
> >> > > You should create your own proto file and compile that with the
> Google
> >> > > Protocol Buffer compiler. Take a look at the
> SingleColumnValueFilter's
> >> > > code:
> >> > >
> >> >
> >>
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java#L327
> >> > >
> >> > > You will need to override `public byte[] toByteArray()` and `public
> >> > static
> >> > > Filter parseFrom(byte[] pbBytes)`. The output of toByteArray()
> should
> >> be
> >> > > the byte[] from serializing with the protocol buffer. This will also
> >> be
> >> > the
> >> > > input to parseFrom(byte[]), which is called using reflection on the
> >> > > server-side to instantiate your custom filter and use it.
> >> > >
> >> > > On Sun, Oct 19, 2014 at 11:31 AM, Matt K <matvey1414@gmail.com
> >> > <javascript:;>> wrote:
> >> > >
> >> > >> Anyone?
> >> > >>
> >> > >> On Thursday, October 16, 2014, Matt K <matvey1414@gmail.com
> >> > <javascript:;>> wrote:
> >> > >>
> >> > >> > Hi, can anyone help with above? Feels like I'm missing something
> >> > >> obvious.
> >> > >> >
> >> > >> > On Wednesday, October 15, 2014, Nishanth S <
> >> nishanth.2884@gmail.com
> >> > <javascript:;>
> >> > >> > <javascript:_e(%7B%7D,'cvml','nishanth.2884@gmail.com
> >> <javascript:;>');>>
> >> > wrote:
> >> > >> >
> >> > >> >> Thanks Ted .I will take a look.
> >> > >> >>
> >> > >> >> -Nishanth
> >> > >> >>
> >> > >> >> On Wed, Oct 15, 2014 at 3:43 PM, Ted Yu <yuzhihong@gmail.com
> >> > <javascript:;>> wrote:
> >> > >> >>
> >> > >> >> > Nishanth:
> >> > >> >> > Good question.
> >> > >> >> >
> >> > >> >> > As a general coding guide, writing unit test is always a good
> >> > start.
> >> > >> >> Using
> >> > >> >> > Matt's case as an example, take a look at TestPrefixFilter.
> >> > >> >> >
> >> > >> >> > There're various unit tests for Filters in hbase code.
> >> > >> >> >
> >> > >> >> > Cheers
> >> > >> >> >
> >> > >> >> > On Wed, Oct 15, 2014 at 2:30 PM, Nishanth S <
> >> > nishanth.2884@gmail.com <javascript:;>
> >> > >> >
> >> > >> >> > wrote:
> >> > >> >> >
> >> > >> >> > > Hi Ted ,
> >> > >> >> > > Since I am also working on similar thing is there a way we
> >> can
> >> > >> first
> >> > >> >> > test
> >> > >> >> > > the filter on client side?.You know what I mean without
> >> > disrupting
> >> > >> >> > others
> >> > >> >> > > who are using the same cluster for other work?
> >> > >> >> > >
> >> > >> >> > > Thanks,
> >> > >> >> > > Nishanth
> >> > >> >> > >
> >> > >> >> > > On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu <
> yuzhihong@gmail.com
> >> > <javascript:;>>
> >> > >> wrote:
> >> > >> >> > >
> >> > >> >> > > > bq. Or create a new file, compile it into ...
> >> > >> >> > > >
> >> > >> >> > > > You should go with the above approach.
> >> > >> >> > > >
> >> > >> >> > > > On Wed, Oct 15, 2014 at 2:08 PM, Matt K <
> >> matvey1414@gmail.com
> >> > <javascript:;>>
> >> > >> >> wrote:
> >> > >> >> > > >
> >> > >> >> > > > > Hi all,
> >> > >> >> > > > >
> >> > >> >> > > > > I'm trying to get a custom filter to work on HBase 0.96.
> >> > After
> >> > >> >> some
> >> > >> >> > > > > searching, I found that starting from 0.96, the
> >> implementer
> >> > is
> >> > >> >> > required
> >> > >> >> > > > to
> >> > >> >> > > > > implement "toByteArray" and "parseFrom" methods, using
> >> > Protocol
> >> > >> >> > > Buffers.
> >> > >> >> > > > > But I'm having trouble with the "how".
> >> > >> >> > > > >
> >> > >> >> > > > > The "proto" file for the existing filters is located
> here:
> >> > >> >> > > > >
> >> > >> >> > > > >
> >> > >> >> > > >
> >> > >> >> > >
> >> > >> >> >
> >> > >> >>
> >> > >>
> >> >
> >>
> https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/protobuf/Filter.proto
> >> > >> >> > > > >
> >> > >> >> > > > > Am I supposed to modify that file? Or create a new file,
> >> > >> compile
> >> > >> >> it
> >> > >> >> > > into
> >> > >> >> > > > > Java, and package it up with the filter?
> >> > >> >> > > > >
> >> > >> >> > > > > In the meantime, I've taken a shortcut that's not
> working.
> >> > >> Here's
> >> > >> >> my
> >> > >> >> > > > code:
> >> > >> >> > > > > http://pastebin.com/iHFKu9Xz
> >> > >> >> > > > >
> >> > >> >> > > > > I'm using "PrefixFilter", which comes with HBase, since
> >> I'm
> >> > >> also
> >> > >> >> > > > filtering
> >> > >> >> > > > > by "prefix". However, that errors out with the
> following:
> >> > >> >> > > > > http://pastebin.com/zBg47p6Z
> >> > >> >> > > > >
> >> > >> >> > > > > Thanks in advance for helping!
> >> > >> >> > > > >
> >> > >> >> > > > > -Matt
> >> > >> >> > > > >
> >> > >> >> > > >
> >> > >> >> > >
> >> > >> >> >
> >> > >> >>
> >> > >> >
> >> > >> >
> >> > >> > --
> >> > >> > www.calcmachine.com - easy online calculator.
> >> > >> >
> >> > >>
> >> > >>
> >> > >> --
> >> > >> www.calcmachine.com - easy online calculator.
> >> > >>
> >> > >
> >> > >
> >> >
> >>
> >>
> >> --
> >> www.calcmachine.com - easy online calculator.
> >>
> >
> >
>
Re: custom filter on hbase 0.96
Posted by Kevin <ke...@gmail.com>.
Also, if you do end up using dynamic loading, you'll need a way to version
your filters because the RS will not reload a JAR if it changes.
On Tue, Oct 21, 2014 at 9:46 PM, Kevin <ke...@gmail.com> wrote:
> I haven't tried dynamic loading of filters on RS, but I know it does
> exist. See https://issues.apache.org/jira/browse/HBASE-9301.
>
> If you still can't get it to work, then I suggest distributing your
> filters to the RS and restart them. Let us know how everything works out.
>
> On Tue, Oct 21, 2014 at 9:02 PM, Matt K <ma...@gmail.com> wrote:
>
>> Thanks Kevin!
>>
>> I was under impression, probably mistakingly, that as of 0.96 placing
>> the filter on hdfs under hbase lib directory is sufficient and RS should
>> load the filter dynamically from hdfs. Is that not the case?
>>
>> On Tuesday, October 21, 2014, Kevin <ke...@gmail.com> wrote:
>>
>> > BTW, the error looks like you didn't distribute your custom filter to
>> your
>> > region servers.
>> >
>> > On Tue, Oct 21, 2014 at 1:34 PM, Kevin <kevin.macksamie@gmail.com
>> > <javascript:;>> wrote:
>> >
>> > > Matt,
>> > >
>> > > You should create your own proto file and compile that with the Google
>> > > Protocol Buffer compiler. Take a look at the SingleColumnValueFilter's
>> > > code:
>> > >
>> >
>> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java#L327
>> > >
>> > > You will need to override `public byte[] toByteArray()` and `public
>> > static
>> > > Filter parseFrom(byte[] pbBytes)`. The output of toByteArray() should
>> be
>> > > the byte[] from serializing with the protocol buffer. This will also
>> be
>> > the
>> > > input to parseFrom(byte[]), which is called using reflection on the
>> > > server-side to instantiate your custom filter and use it.
>> > >
>> > > On Sun, Oct 19, 2014 at 11:31 AM, Matt K <matvey1414@gmail.com
>> > <javascript:;>> wrote:
>> > >
>> > >> Anyone?
>> > >>
>> > >> On Thursday, October 16, 2014, Matt K <matvey1414@gmail.com
>> > <javascript:;>> wrote:
>> > >>
>> > >> > Hi, can anyone help with above? Feels like I'm missing something
>> > >> obvious.
>> > >> >
>> > >> > On Wednesday, October 15, 2014, Nishanth S <
>> nishanth.2884@gmail.com
>> > <javascript:;>
>> > >> > <javascript:_e(%7B%7D,'cvml','nishanth.2884@gmail.com
>> <javascript:;>');>>
>> > wrote:
>> > >> >
>> > >> >> Thanks Ted .I will take a look.
>> > >> >>
>> > >> >> -Nishanth
>> > >> >>
>> > >> >> On Wed, Oct 15, 2014 at 3:43 PM, Ted Yu <yuzhihong@gmail.com
>> > <javascript:;>> wrote:
>> > >> >>
>> > >> >> > Nishanth:
>> > >> >> > Good question.
>> > >> >> >
>> > >> >> > As a general coding guide, writing unit test is always a good
>> > start.
>> > >> >> Using
>> > >> >> > Matt's case as an example, take a look at TestPrefixFilter.
>> > >> >> >
>> > >> >> > There're various unit tests for Filters in hbase code.
>> > >> >> >
>> > >> >> > Cheers
>> > >> >> >
>> > >> >> > On Wed, Oct 15, 2014 at 2:30 PM, Nishanth S <
>> > nishanth.2884@gmail.com <javascript:;>
>> > >> >
>> > >> >> > wrote:
>> > >> >> >
>> > >> >> > > Hi Ted ,
>> > >> >> > > Since I am also working on similar thing is there a way we
>> can
>> > >> first
>> > >> >> > test
>> > >> >> > > the filter on client side?.You know what I mean without
>> > disrupting
>> > >> >> > others
>> > >> >> > > who are using the same cluster for other work?
>> > >> >> > >
>> > >> >> > > Thanks,
>> > >> >> > > Nishanth
>> > >> >> > >
>> > >> >> > > On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu <yuzhihong@gmail.com
>> > <javascript:;>>
>> > >> wrote:
>> > >> >> > >
>> > >> >> > > > bq. Or create a new file, compile it into ...
>> > >> >> > > >
>> > >> >> > > > You should go with the above approach.
>> > >> >> > > >
>> > >> >> > > > On Wed, Oct 15, 2014 at 2:08 PM, Matt K <
>> matvey1414@gmail.com
>> > <javascript:;>>
>> > >> >> wrote:
>> > >> >> > > >
>> > >> >> > > > > Hi all,
>> > >> >> > > > >
>> > >> >> > > > > I'm trying to get a custom filter to work on HBase 0.96.
>> > After
>> > >> >> some
>> > >> >> > > > > searching, I found that starting from 0.96, the
>> implementer
>> > is
>> > >> >> > required
>> > >> >> > > > to
>> > >> >> > > > > implement "toByteArray" and "parseFrom" methods, using
>> > Protocol
>> > >> >> > > Buffers.
>> > >> >> > > > > But I'm having trouble with the "how".
>> > >> >> > > > >
>> > >> >> > > > > The "proto" file for the existing filters is located here:
>> > >> >> > > > >
>> > >> >> > > > >
>> > >> >> > > >
>> > >> >> > >
>> > >> >> >
>> > >> >>
>> > >>
>> >
>> https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/protobuf/Filter.proto
>> > >> >> > > > >
>> > >> >> > > > > Am I supposed to modify that file? Or create a new file,
>> > >> compile
>> > >> >> it
>> > >> >> > > into
>> > >> >> > > > > Java, and package it up with the filter?
>> > >> >> > > > >
>> > >> >> > > > > In the meantime, I've taken a shortcut that's not working.
>> > >> Here's
>> > >> >> my
>> > >> >> > > > code:
>> > >> >> > > > > http://pastebin.com/iHFKu9Xz
>> > >> >> > > > >
>> > >> >> > > > > I'm using "PrefixFilter", which comes with HBase, since
>> I'm
>> > >> also
>> > >> >> > > > filtering
>> > >> >> > > > > by "prefix". However, that errors out with the following:
>> > >> >> > > > > http://pastebin.com/zBg47p6Z
>> > >> >> > > > >
>> > >> >> > > > > Thanks in advance for helping!
>> > >> >> > > > >
>> > >> >> > > > > -Matt
>> > >> >> > > > >
>> > >> >> > > >
>> > >> >> > >
>> > >> >> >
>> > >> >>
>> > >> >
>> > >> >
>> > >> > --
>> > >> > www.calcmachine.com - easy online calculator.
>> > >> >
>> > >>
>> > >>
>> > >> --
>> > >> www.calcmachine.com - easy online calculator.
>> > >>
>> > >
>> > >
>> >
>>
>>
>> --
>> www.calcmachine.com - easy online calculator.
>>
>
>
Re: custom filter on hbase 0.96
Posted by Kevin <ke...@gmail.com>.
I haven't tried dynamic loading of filters on RS, but I know it does exist.
See https://issues.apache.org/jira/browse/HBASE-9301.
If you still can't get it to work, then I suggest distributing your filters
to the RS and restart them. Let us know how everything works out.
On Tue, Oct 21, 2014 at 9:02 PM, Matt K <ma...@gmail.com> wrote:
> Thanks Kevin!
>
> I was under impression, probably mistakingly, that as of 0.96 placing
> the filter on hdfs under hbase lib directory is sufficient and RS should
> load the filter dynamically from hdfs. Is that not the case?
>
> On Tuesday, October 21, 2014, Kevin <ke...@gmail.com> wrote:
>
> > BTW, the error looks like you didn't distribute your custom filter to
> your
> > region servers.
> >
> > On Tue, Oct 21, 2014 at 1:34 PM, Kevin <kevin.macksamie@gmail.com
> > <javascript:;>> wrote:
> >
> > > Matt,
> > >
> > > You should create your own proto file and compile that with the Google
> > > Protocol Buffer compiler. Take a look at the SingleColumnValueFilter's
> > > code:
> > >
> >
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java#L327
> > >
> > > You will need to override `public byte[] toByteArray()` and `public
> > static
> > > Filter parseFrom(byte[] pbBytes)`. The output of toByteArray() should
> be
> > > the byte[] from serializing with the protocol buffer. This will also be
> > the
> > > input to parseFrom(byte[]), which is called using reflection on the
> > > server-side to instantiate your custom filter and use it.
> > >
> > > On Sun, Oct 19, 2014 at 11:31 AM, Matt K <matvey1414@gmail.com
> > <javascript:;>> wrote:
> > >
> > >> Anyone?
> > >>
> > >> On Thursday, October 16, 2014, Matt K <matvey1414@gmail.com
> > <javascript:;>> wrote:
> > >>
> > >> > Hi, can anyone help with above? Feels like I'm missing something
> > >> obvious.
> > >> >
> > >> > On Wednesday, October 15, 2014, Nishanth S <nishanth.2884@gmail.com
> > <javascript:;>
> > >> > <javascript:_e(%7B%7D,'cvml','nishanth.2884@gmail.com
> <javascript:;>');>>
> > wrote:
> > >> >
> > >> >> Thanks Ted .I will take a look.
> > >> >>
> > >> >> -Nishanth
> > >> >>
> > >> >> On Wed, Oct 15, 2014 at 3:43 PM, Ted Yu <yuzhihong@gmail.com
> > <javascript:;>> wrote:
> > >> >>
> > >> >> > Nishanth:
> > >> >> > Good question.
> > >> >> >
> > >> >> > As a general coding guide, writing unit test is always a good
> > start.
> > >> >> Using
> > >> >> > Matt's case as an example, take a look at TestPrefixFilter.
> > >> >> >
> > >> >> > There're various unit tests for Filters in hbase code.
> > >> >> >
> > >> >> > Cheers
> > >> >> >
> > >> >> > On Wed, Oct 15, 2014 at 2:30 PM, Nishanth S <
> > nishanth.2884@gmail.com <javascript:;>
> > >> >
> > >> >> > wrote:
> > >> >> >
> > >> >> > > Hi Ted ,
> > >> >> > > Since I am also working on similar thing is there a way we can
> > >> first
> > >> >> > test
> > >> >> > > the filter on client side?.You know what I mean without
> > disrupting
> > >> >> > others
> > >> >> > > who are using the same cluster for other work?
> > >> >> > >
> > >> >> > > Thanks,
> > >> >> > > Nishanth
> > >> >> > >
> > >> >> > > On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu <yuzhihong@gmail.com
> > <javascript:;>>
> > >> wrote:
> > >> >> > >
> > >> >> > > > bq. Or create a new file, compile it into ...
> > >> >> > > >
> > >> >> > > > You should go with the above approach.
> > >> >> > > >
> > >> >> > > > On Wed, Oct 15, 2014 at 2:08 PM, Matt K <
> matvey1414@gmail.com
> > <javascript:;>>
> > >> >> wrote:
> > >> >> > > >
> > >> >> > > > > Hi all,
> > >> >> > > > >
> > >> >> > > > > I'm trying to get a custom filter to work on HBase 0.96.
> > After
> > >> >> some
> > >> >> > > > > searching, I found that starting from 0.96, the implementer
> > is
> > >> >> > required
> > >> >> > > > to
> > >> >> > > > > implement "toByteArray" and "parseFrom" methods, using
> > Protocol
> > >> >> > > Buffers.
> > >> >> > > > > But I'm having trouble with the "how".
> > >> >> > > > >
> > >> >> > > > > The "proto" file for the existing filters is located here:
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
> https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/protobuf/Filter.proto
> > >> >> > > > >
> > >> >> > > > > Am I supposed to modify that file? Or create a new file,
> > >> compile
> > >> >> it
> > >> >> > > into
> > >> >> > > > > Java, and package it up with the filter?
> > >> >> > > > >
> > >> >> > > > > In the meantime, I've taken a shortcut that's not working.
> > >> Here's
> > >> >> my
> > >> >> > > > code:
> > >> >> > > > > http://pastebin.com/iHFKu9Xz
> > >> >> > > > >
> > >> >> > > > > I'm using "PrefixFilter", which comes with HBase, since I'm
> > >> also
> > >> >> > > > filtering
> > >> >> > > > > by "prefix". However, that errors out with the following:
> > >> >> > > > > http://pastebin.com/zBg47p6Z
> > >> >> > > > >
> > >> >> > > > > Thanks in advance for helping!
> > >> >> > > > >
> > >> >> > > > > -Matt
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >> >
> > >> >
> > >> > --
> > >> > www.calcmachine.com - easy online calculator.
> > >> >
> > >>
> > >>
> > >> --
> > >> www.calcmachine.com - easy online calculator.
> > >>
> > >
> > >
> >
>
>
> --
> www.calcmachine.com - easy online calculator.
>
Re: custom filter on hbase 0.96
Posted by Anil Gupta <an...@gmail.com>.
To deploy a custom filter. You just need to add the jar file of custom filter in classpath of HBase on each region server. Then do a restart of each region server.
HTH,
Anil
Sent from my iPhone
> On Apr 1, 2015, at 1:48 AM, Jeroen Hoek <je...@lable.org> wrote:
>
> I too would love to find out more about distributing custom Filter
> classes. Any pointers would be appreciated.
>
>> On 18-03-15 04:04, Marc Sturm wrote:
>> Matt, Were you able to successfully distribute your custom Filter?
>> Yes, can you tell us how you did it? Thanks Marc
>>
>>> -----Original Message----- From: Matt K
>>> [mailto:matvey1414@gmail.com] Sent: Tuesday, October 21, 2014 9:02
>>> PM To: user@hbase.apache.org Subject: Re: custom filter on hbase
>>> 0.96
>>>
>>> Thanks Kevin!
>>>
>>> I was under impression, probably mistakingly, that as of 0.96
>>> placing the filter on hdfs under hbase lib directory is sufficient
>>> and RS should load the filter dynamically from hdfs. Is that not
>>> the case?
>
> --
> Kind regards,
>
> Jeroen Hoek
> http://lable.org — KvK № 55984037 — BTW № NL8519.32.411.B.01
>
>
Re: custom filter on hbase 0.96
Posted by Jeroen Hoek <je...@lable.org>.
I too would love to find out more about distributing custom Filter
classes. Any pointers would be appreciated.
On 18-03-15 04:04, Marc Sturm wrote:
> Matt, Were you able to successfully distribute your custom Filter?
> Yes, can you tell us how you did it? Thanks Marc
>
>> -----Original Message----- From: Matt K
>> [mailto:matvey1414@gmail.com] Sent: Tuesday, October 21, 2014 9:02
>> PM To: user@hbase.apache.org Subject: Re: custom filter on hbase
>> 0.96
>>
>> Thanks Kevin!
>>
>> I was under impression, probably mistakingly, that as of 0.96
>> placing the filter on hdfs under hbase lib directory is sufficient
>> and RS should load the filter dynamically from hdfs. Is that not
>> the case?
--
Kind regards,
Jeroen Hoek
http://lable.org — KvK № 55984037 — BTW № NL8519.32.411.B.01
RE: custom filter on hbase 0.96
Posted by Marc Sturm <ma...@nyp.org>.
Matt,
Were you able to successfully distribute your custom Filter? Yes, can you tell us how you did it?
Thanks
Marc
-----Original Message-----
From: Matt K [mailto:matvey1414@gmail.com]
Sent: Tuesday, October 21, 2014 9:02 PM
To: user@hbase.apache.org
Subject: Re: custom filter on hbase 0.96
Thanks Kevin!
I was under impression, probably mistakingly, that as of 0.96 placing the filter on hdfs under hbase lib directory is sufficient and RS should load the filter dynamically from hdfs. Is that not the case?
On Tuesday, October 21, 2014, Kevin <ke...@gmail.com> wrote:
> BTW, the error looks like you didn't distribute your custom filter to
> your region servers.
>
> On Tue, Oct 21, 2014 at 1:34 PM, Kevin <kevin.macksamie@gmail.com
> <javascript:;>> wrote:
>
> > Matt,
> >
> > You should create your own proto file and compile that with the
> > Google Protocol Buffer compiler. Take a look at the
> > SingleColumnValueFilter's
> > code:
> >
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java
> /org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java#L327
> >
> > You will need to override `public byte[] toByteArray()` and `public
> static
> > Filter parseFrom(byte[] pbBytes)`. The output of toByteArray()
> > should be the byte[] from serializing with the protocol buffer. This
> > will also be
> the
> > input to parseFrom(byte[]), which is called using reflection on the
> > server-side to instantiate your custom filter and use it.
> >
> > On Sun, Oct 19, 2014 at 11:31 AM, Matt K <matvey1414@gmail.com
> <javascript:;>> wrote:
> >
> >> Anyone?
> >>
> >> On Thursday, October 16, 2014, Matt K <matvey1414@gmail.com
> <javascript:;>> wrote:
> >>
> >> > Hi, can anyone help with above? Feels like I'm missing something
> >> obvious.
> >> >
> >> > On Wednesday, October 15, 2014, Nishanth S
> >> > <nishanth.2884@gmail.com
> <javascript:;>
> >> > <javascript:_e(%7B%7D,'cvml','nishanth.2884@gmail.com
> >> > <javascript:;>');>>
> wrote:
> >> >
> >> >> Thanks Ted .I will take a look.
> >> >>
> >> >> -Nishanth
> >> >>
> >> >> On Wed, Oct 15, 2014 at 3:43 PM, Ted Yu <yuzhihong@gmail.com
> <javascript:;>> wrote:
> >> >>
> >> >> > Nishanth:
> >> >> > Good question.
> >> >> >
> >> >> > As a general coding guide, writing unit test is always a good
> start.
> >> >> Using
> >> >> > Matt's case as an example, take a look at TestPrefixFilter.
> >> >> >
> >> >> > There're various unit tests for Filters in hbase code.
> >> >> >
> >> >> > Cheers
> >> >> >
> >> >> > On Wed, Oct 15, 2014 at 2:30 PM, Nishanth S <
> nishanth.2884@gmail.com <javascript:;>
> >> >
> >> >> > wrote:
> >> >> >
> >> >> > > Hi Ted ,
> >> >> > > Since I am also working on similar thing is there a way we
> >> >> > > can
> >> first
> >> >> > test
> >> >> > > the filter on client side?.You know what I mean without
> disrupting
> >> >> > others
> >> >> > > who are using the same cluster for other work?
> >> >> > >
> >> >> > > Thanks,
> >> >> > > Nishanth
> >> >> > >
> >> >> > > On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu <yuzhihong@gmail.com
> <javascript:;>>
> >> wrote:
> >> >> > >
> >> >> > > > bq. Or create a new file, compile it into ...
> >> >> > > >
> >> >> > > > You should go with the above approach.
> >> >> > > >
> >> >> > > > On Wed, Oct 15, 2014 at 2:08 PM, Matt K
> >> >> > > > <matvey1414@gmail.com
> <javascript:;>>
> >> >> wrote:
> >> >> > > >
> >> >> > > > > Hi all,
> >> >> > > > >
> >> >> > > > > I'm trying to get a custom filter to work on HBase 0.96.
> After
> >> >> some
> >> >> > > > > searching, I found that starting from 0.96, the
> >> >> > > > > implementer
> is
> >> >> > required
> >> >> > > > to
> >> >> > > > > implement "toByteArray" and "parseFrom" methods, using
> Protocol
> >> >> > > Buffers.
> >> >> > > > > But I'm having trouble with the "how".
> >> >> > > > >
> >> >> > > > > The "proto" file for the existing filters is located here:
> >> >> > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >>
> https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/pr
> otobuf/Filter.proto
> >> >> > > > >
> >> >> > > > > Am I supposed to modify that file? Or create a new file,
> >> compile
> >> >> it
> >> >> > > into
> >> >> > > > > Java, and package it up with the filter?
> >> >> > > > >
> >> >> > > > > In the meantime, I've taken a shortcut that's not working.
> >> Here's
> >> >> my
> >> >> > > > code:
> >> >> > > > > http://pastebin.com/iHFKu9Xz
> >> >> > > > >
> >> >> > > > > I'm using "PrefixFilter", which comes with HBase, since
> >> >> > > > > I'm
> >> also
> >> >> > > > filtering
> >> >> > > > > by "prefix". However, that errors out with the following:
> >> >> > > > > http://pastebin.com/zBg47p6Z
> >> >> > > > >
> >> >> > > > > Thanks in advance for helping!
> >> >> > > > >
> >> >> > > > > -Matt
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >> >
> >> >
> >> > --
> >> > www.calcmachine.com - easy online calculator.
> >> >
> >>
> >>
> >> --
> >> www.calcmachine.com - easy online calculator.
> >>
> >
> >
>
--
www.calcmachine.com - easy online calculator.
This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
Re: custom filter on hbase 0.96
Posted by Matt K <ma...@gmail.com>.
Thanks Kevin!
I was under impression, probably mistakingly, that as of 0.96 placing
the filter on hdfs under hbase lib directory is sufficient and RS should
load the filter dynamically from hdfs. Is that not the case?
On Tuesday, October 21, 2014, Kevin <ke...@gmail.com> wrote:
> BTW, the error looks like you didn't distribute your custom filter to your
> region servers.
>
> On Tue, Oct 21, 2014 at 1:34 PM, Kevin <kevin.macksamie@gmail.com
> <javascript:;>> wrote:
>
> > Matt,
> >
> > You should create your own proto file and compile that with the Google
> > Protocol Buffer compiler. Take a look at the SingleColumnValueFilter's
> > code:
> >
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java#L327
> >
> > You will need to override `public byte[] toByteArray()` and `public
> static
> > Filter parseFrom(byte[] pbBytes)`. The output of toByteArray() should be
> > the byte[] from serializing with the protocol buffer. This will also be
> the
> > input to parseFrom(byte[]), which is called using reflection on the
> > server-side to instantiate your custom filter and use it.
> >
> > On Sun, Oct 19, 2014 at 11:31 AM, Matt K <matvey1414@gmail.com
> <javascript:;>> wrote:
> >
> >> Anyone?
> >>
> >> On Thursday, October 16, 2014, Matt K <matvey1414@gmail.com
> <javascript:;>> wrote:
> >>
> >> > Hi, can anyone help with above? Feels like I'm missing something
> >> obvious.
> >> >
> >> > On Wednesday, October 15, 2014, Nishanth S <nishanth.2884@gmail.com
> <javascript:;>
> >> > <javascript:_e(%7B%7D,'cvml','nishanth.2884@gmail.com <javascript:;>');>>
> wrote:
> >> >
> >> >> Thanks Ted .I will take a look.
> >> >>
> >> >> -Nishanth
> >> >>
> >> >> On Wed, Oct 15, 2014 at 3:43 PM, Ted Yu <yuzhihong@gmail.com
> <javascript:;>> wrote:
> >> >>
> >> >> > Nishanth:
> >> >> > Good question.
> >> >> >
> >> >> > As a general coding guide, writing unit test is always a good
> start.
> >> >> Using
> >> >> > Matt's case as an example, take a look at TestPrefixFilter.
> >> >> >
> >> >> > There're various unit tests for Filters in hbase code.
> >> >> >
> >> >> > Cheers
> >> >> >
> >> >> > On Wed, Oct 15, 2014 at 2:30 PM, Nishanth S <
> nishanth.2884@gmail.com <javascript:;>
> >> >
> >> >> > wrote:
> >> >> >
> >> >> > > Hi Ted ,
> >> >> > > Since I am also working on similar thing is there a way we can
> >> first
> >> >> > test
> >> >> > > the filter on client side?.You know what I mean without
> disrupting
> >> >> > others
> >> >> > > who are using the same cluster for other work?
> >> >> > >
> >> >> > > Thanks,
> >> >> > > Nishanth
> >> >> > >
> >> >> > > On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu <yuzhihong@gmail.com
> <javascript:;>>
> >> wrote:
> >> >> > >
> >> >> > > > bq. Or create a new file, compile it into ...
> >> >> > > >
> >> >> > > > You should go with the above approach.
> >> >> > > >
> >> >> > > > On Wed, Oct 15, 2014 at 2:08 PM, Matt K <matvey1414@gmail.com
> <javascript:;>>
> >> >> wrote:
> >> >> > > >
> >> >> > > > > Hi all,
> >> >> > > > >
> >> >> > > > > I'm trying to get a custom filter to work on HBase 0.96.
> After
> >> >> some
> >> >> > > > > searching, I found that starting from 0.96, the implementer
> is
> >> >> > required
> >> >> > > > to
> >> >> > > > > implement "toByteArray" and "parseFrom" methods, using
> Protocol
> >> >> > > Buffers.
> >> >> > > > > But I'm having trouble with the "how".
> >> >> > > > >
> >> >> > > > > The "proto" file for the existing filters is located here:
> >> >> > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >>
> https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/protobuf/Filter.proto
> >> >> > > > >
> >> >> > > > > Am I supposed to modify that file? Or create a new file,
> >> compile
> >> >> it
> >> >> > > into
> >> >> > > > > Java, and package it up with the filter?
> >> >> > > > >
> >> >> > > > > In the meantime, I've taken a shortcut that's not working.
> >> Here's
> >> >> my
> >> >> > > > code:
> >> >> > > > > http://pastebin.com/iHFKu9Xz
> >> >> > > > >
> >> >> > > > > I'm using "PrefixFilter", which comes with HBase, since I'm
> >> also
> >> >> > > > filtering
> >> >> > > > > by "prefix". However, that errors out with the following:
> >> >> > > > > http://pastebin.com/zBg47p6Z
> >> >> > > > >
> >> >> > > > > Thanks in advance for helping!
> >> >> > > > >
> >> >> > > > > -Matt
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >> >
> >> >
> >> > --
> >> > www.calcmachine.com - easy online calculator.
> >> >
> >>
> >>
> >> --
> >> www.calcmachine.com - easy online calculator.
> >>
> >
> >
>
--
www.calcmachine.com - easy online calculator.
Re: custom filter on hbase 0.96
Posted by Kevin <ke...@gmail.com>.
BTW, the error looks like you didn't distribute your custom filter to your
region servers.
On Tue, Oct 21, 2014 at 1:34 PM, Kevin <ke...@gmail.com> wrote:
> Matt,
>
> You should create your own proto file and compile that with the Google
> Protocol Buffer compiler. Take a look at the SingleColumnValueFilter's
> code:
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java#L327
>
> You will need to override `public byte[] toByteArray()` and `public static
> Filter parseFrom(byte[] pbBytes)`. The output of toByteArray() should be
> the byte[] from serializing with the protocol buffer. This will also be the
> input to parseFrom(byte[]), which is called using reflection on the
> server-side to instantiate your custom filter and use it.
>
> On Sun, Oct 19, 2014 at 11:31 AM, Matt K <ma...@gmail.com> wrote:
>
>> Anyone?
>>
>> On Thursday, October 16, 2014, Matt K <ma...@gmail.com> wrote:
>>
>> > Hi, can anyone help with above? Feels like I'm missing something
>> obvious.
>> >
>> > On Wednesday, October 15, 2014, Nishanth S <nishanth.2884@gmail.com
>> > <javascript:_e(%7B%7D,'cvml','nishanth.2884@gmail.com');>> wrote:
>> >
>> >> Thanks Ted .I will take a look.
>> >>
>> >> -Nishanth
>> >>
>> >> On Wed, Oct 15, 2014 at 3:43 PM, Ted Yu <yu...@gmail.com> wrote:
>> >>
>> >> > Nishanth:
>> >> > Good question.
>> >> >
>> >> > As a general coding guide, writing unit test is always a good start.
>> >> Using
>> >> > Matt's case as an example, take a look at TestPrefixFilter.
>> >> >
>> >> > There're various unit tests for Filters in hbase code.
>> >> >
>> >> > Cheers
>> >> >
>> >> > On Wed, Oct 15, 2014 at 2:30 PM, Nishanth S <nishanth.2884@gmail.com
>> >
>> >> > wrote:
>> >> >
>> >> > > Hi Ted ,
>> >> > > Since I am also working on similar thing is there a way we can
>> first
>> >> > test
>> >> > > the filter on client side?.You know what I mean without disrupting
>> >> > others
>> >> > > who are using the same cluster for other work?
>> >> > >
>> >> > > Thanks,
>> >> > > Nishanth
>> >> > >
>> >> > > On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu <yu...@gmail.com>
>> wrote:
>> >> > >
>> >> > > > bq. Or create a new file, compile it into ...
>> >> > > >
>> >> > > > You should go with the above approach.
>> >> > > >
>> >> > > > On Wed, Oct 15, 2014 at 2:08 PM, Matt K <ma...@gmail.com>
>> >> wrote:
>> >> > > >
>> >> > > > > Hi all,
>> >> > > > >
>> >> > > > > I'm trying to get a custom filter to work on HBase 0.96. After
>> >> some
>> >> > > > > searching, I found that starting from 0.96, the implementer is
>> >> > required
>> >> > > > to
>> >> > > > > implement "toByteArray" and "parseFrom" methods, using Protocol
>> >> > > Buffers.
>> >> > > > > But I'm having trouble with the "how".
>> >> > > > >
>> >> > > > > The "proto" file for the existing filters is located here:
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/protobuf/Filter.proto
>> >> > > > >
>> >> > > > > Am I supposed to modify that file? Or create a new file,
>> compile
>> >> it
>> >> > > into
>> >> > > > > Java, and package it up with the filter?
>> >> > > > >
>> >> > > > > In the meantime, I've taken a shortcut that's not working.
>> Here's
>> >> my
>> >> > > > code:
>> >> > > > > http://pastebin.com/iHFKu9Xz
>> >> > > > >
>> >> > > > > I'm using "PrefixFilter", which comes with HBase, since I'm
>> also
>> >> > > > filtering
>> >> > > > > by "prefix". However, that errors out with the following:
>> >> > > > > http://pastebin.com/zBg47p6Z
>> >> > > > >
>> >> > > > > Thanks in advance for helping!
>> >> > > > >
>> >> > > > > -Matt
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>> >
>> > --
>> > www.calcmachine.com - easy online calculator.
>> >
>>
>>
>> --
>> www.calcmachine.com - easy online calculator.
>>
>
>
Re: custom filter on hbase 0.96
Posted by Kevin <ke...@gmail.com>.
Matt,
You should create your own proto file and compile that with the Google
Protocol Buffer compiler. Take a look at the SingleColumnValueFilter's
code:
https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java#L327
You will need to override `public byte[] toByteArray()` and `public static
Filter parseFrom(byte[] pbBytes)`. The output of toByteArray() should be
the byte[] from serializing with the protocol buffer. This will also be the
input to parseFrom(byte[]), which is called using reflection on the
server-side to instantiate your custom filter and use it.
On Sun, Oct 19, 2014 at 11:31 AM, Matt K <ma...@gmail.com> wrote:
> Anyone?
>
> On Thursday, October 16, 2014, Matt K <ma...@gmail.com> wrote:
>
> > Hi, can anyone help with above? Feels like I'm missing something obvious.
> >
> > On Wednesday, October 15, 2014, Nishanth S <nishanth.2884@gmail.com
> > <javascript:_e(%7B%7D,'cvml','nishanth.2884@gmail.com');>> wrote:
> >
> >> Thanks Ted .I will take a look.
> >>
> >> -Nishanth
> >>
> >> On Wed, Oct 15, 2014 at 3:43 PM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >> > Nishanth:
> >> > Good question.
> >> >
> >> > As a general coding guide, writing unit test is always a good start.
> >> Using
> >> > Matt's case as an example, take a look at TestPrefixFilter.
> >> >
> >> > There're various unit tests for Filters in hbase code.
> >> >
> >> > Cheers
> >> >
> >> > On Wed, Oct 15, 2014 at 2:30 PM, Nishanth S <ni...@gmail.com>
> >> > wrote:
> >> >
> >> > > Hi Ted ,
> >> > > Since I am also working on similar thing is there a way we can
> first
> >> > test
> >> > > the filter on client side?.You know what I mean without disrupting
> >> > others
> >> > > who are using the same cluster for other work?
> >> > >
> >> > > Thanks,
> >> > > Nishanth
> >> > >
> >> > > On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu <yu...@gmail.com>
> wrote:
> >> > >
> >> > > > bq. Or create a new file, compile it into ...
> >> > > >
> >> > > > You should go with the above approach.
> >> > > >
> >> > > > On Wed, Oct 15, 2014 at 2:08 PM, Matt K <ma...@gmail.com>
> >> wrote:
> >> > > >
> >> > > > > Hi all,
> >> > > > >
> >> > > > > I'm trying to get a custom filter to work on HBase 0.96. After
> >> some
> >> > > > > searching, I found that starting from 0.96, the implementer is
> >> > required
> >> > > > to
> >> > > > > implement "toByteArray" and "parseFrom" methods, using Protocol
> >> > > Buffers.
> >> > > > > But I'm having trouble with the "how".
> >> > > > >
> >> > > > > The "proto" file for the existing filters is located here:
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/protobuf/Filter.proto
> >> > > > >
> >> > > > > Am I supposed to modify that file? Or create a new file, compile
> >> it
> >> > > into
> >> > > > > Java, and package it up with the filter?
> >> > > > >
> >> > > > > In the meantime, I've taken a shortcut that's not working.
> Here's
> >> my
> >> > > > code:
> >> > > > > http://pastebin.com/iHFKu9Xz
> >> > > > >
> >> > > > > I'm using "PrefixFilter", which comes with HBase, since I'm also
> >> > > > filtering
> >> > > > > by "prefix". However, that errors out with the following:
> >> > > > > http://pastebin.com/zBg47p6Z
> >> > > > >
> >> > > > > Thanks in advance for helping!
> >> > > > >
> >> > > > > -Matt
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
> > --
> > www.calcmachine.com - easy online calculator.
> >
>
>
> --
> www.calcmachine.com - easy online calculator.
>
Re: custom filter on hbase 0.96
Posted by Matt K <ma...@gmail.com>.
Anyone?
On Thursday, October 16, 2014, Matt K <ma...@gmail.com> wrote:
> Hi, can anyone help with above? Feels like I'm missing something obvious.
>
> On Wednesday, October 15, 2014, Nishanth S <nishanth.2884@gmail.com
> <javascript:_e(%7B%7D,'cvml','nishanth.2884@gmail.com');>> wrote:
>
>> Thanks Ted .I will take a look.
>>
>> -Nishanth
>>
>> On Wed, Oct 15, 2014 at 3:43 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>> > Nishanth:
>> > Good question.
>> >
>> > As a general coding guide, writing unit test is always a good start.
>> Using
>> > Matt's case as an example, take a look at TestPrefixFilter.
>> >
>> > There're various unit tests for Filters in hbase code.
>> >
>> > Cheers
>> >
>> > On Wed, Oct 15, 2014 at 2:30 PM, Nishanth S <ni...@gmail.com>
>> > wrote:
>> >
>> > > Hi Ted ,
>> > > Since I am also working on similar thing is there a way we can first
>> > test
>> > > the filter on client side?.You know what I mean without disrupting
>> > others
>> > > who are using the same cluster for other work?
>> > >
>> > > Thanks,
>> > > Nishanth
>> > >
>> > > On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu <yu...@gmail.com> wrote:
>> > >
>> > > > bq. Or create a new file, compile it into ...
>> > > >
>> > > > You should go with the above approach.
>> > > >
>> > > > On Wed, Oct 15, 2014 at 2:08 PM, Matt K <ma...@gmail.com>
>> wrote:
>> > > >
>> > > > > Hi all,
>> > > > >
>> > > > > I'm trying to get a custom filter to work on HBase 0.96. After
>> some
>> > > > > searching, I found that starting from 0.96, the implementer is
>> > required
>> > > > to
>> > > > > implement "toByteArray" and "parseFrom" methods, using Protocol
>> > > Buffers.
>> > > > > But I'm having trouble with the "how".
>> > > > >
>> > > > > The "proto" file for the existing filters is located here:
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/protobuf/Filter.proto
>> > > > >
>> > > > > Am I supposed to modify that file? Or create a new file, compile
>> it
>> > > into
>> > > > > Java, and package it up with the filter?
>> > > > >
>> > > > > In the meantime, I've taken a shortcut that's not working. Here's
>> my
>> > > > code:
>> > > > > http://pastebin.com/iHFKu9Xz
>> > > > >
>> > > > > I'm using "PrefixFilter", which comes with HBase, since I'm also
>> > > > filtering
>> > > > > by "prefix". However, that errors out with the following:
>> > > > > http://pastebin.com/zBg47p6Z
>> > > > >
>> > > > > Thanks in advance for helping!
>> > > > >
>> > > > > -Matt
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
> --
> www.calcmachine.com - easy online calculator.
>
--
www.calcmachine.com - easy online calculator.
Re: custom filter on hbase 0.96
Posted by Matt K <ma...@gmail.com>.
Hi, can anyone help with above? Feels like I'm missing something obvious.
On Wednesday, October 15, 2014, Nishanth S <ni...@gmail.com> wrote:
> Thanks Ted .I will take a look.
>
> -Nishanth
>
> On Wed, Oct 15, 2014 at 3:43 PM, Ted Yu <yuzhihong@gmail.com
> <javascript:;>> wrote:
>
> > Nishanth:
> > Good question.
> >
> > As a general coding guide, writing unit test is always a good start.
> Using
> > Matt's case as an example, take a look at TestPrefixFilter.
> >
> > There're various unit tests for Filters in hbase code.
> >
> > Cheers
> >
> > On Wed, Oct 15, 2014 at 2:30 PM, Nishanth S <nishanth.2884@gmail.com
> <javascript:;>>
> > wrote:
> >
> > > Hi Ted ,
> > > Since I am also working on similar thing is there a way we can first
> > test
> > > the filter on client side?.You know what I mean without disrupting
> > others
> > > who are using the same cluster for other work?
> > >
> > > Thanks,
> > > Nishanth
> > >
> > > On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu <yuzhihong@gmail.com
> <javascript:;>> wrote:
> > >
> > > > bq. Or create a new file, compile it into ...
> > > >
> > > > You should go with the above approach.
> > > >
> > > > On Wed, Oct 15, 2014 at 2:08 PM, Matt K <matvey1414@gmail.com
> <javascript:;>> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'm trying to get a custom filter to work on HBase 0.96. After some
> > > > > searching, I found that starting from 0.96, the implementer is
> > required
> > > > to
> > > > > implement "toByteArray" and "parseFrom" methods, using Protocol
> > > Buffers.
> > > > > But I'm having trouble with the "how".
> > > > >
> > > > > The "proto" file for the existing filters is located here:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/protobuf/Filter.proto
> > > > >
> > > > > Am I supposed to modify that file? Or create a new file, compile it
> > > into
> > > > > Java, and package it up with the filter?
> > > > >
> > > > > In the meantime, I've taken a shortcut that's not working. Here's
> my
> > > > code:
> > > > > http://pastebin.com/iHFKu9Xz
> > > > >
> > > > > I'm using "PrefixFilter", which comes with HBase, since I'm also
> > > > filtering
> > > > > by "prefix". However, that errors out with the following:
> > > > > http://pastebin.com/zBg47p6Z
> > > > >
> > > > > Thanks in advance for helping!
> > > > >
> > > > > -Matt
> > > > >
> > > >
> > >
> >
>
--
www.calcmachine.com - easy online calculator.
Re: custom filter on hbase 0.96
Posted by Nishanth S <ni...@gmail.com>.
Thanks Ted .I will take a look.
-Nishanth
On Wed, Oct 15, 2014 at 3:43 PM, Ted Yu <yu...@gmail.com> wrote:
> Nishanth:
> Good question.
>
> As a general coding guide, writing unit test is always a good start. Using
> Matt's case as an example, take a look at TestPrefixFilter.
>
> There're various unit tests for Filters in hbase code.
>
> Cheers
>
> On Wed, Oct 15, 2014 at 2:30 PM, Nishanth S <ni...@gmail.com>
> wrote:
>
> > Hi Ted ,
> > Since I am also working on similar thing is there a way we can first
> test
> > the filter on client side?.You know what I mean without disrupting
> others
> > who are using the same cluster for other work?
> >
> > Thanks,
> > Nishanth
> >
> > On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > bq. Or create a new file, compile it into ...
> > >
> > > You should go with the above approach.
> > >
> > > On Wed, Oct 15, 2014 at 2:08 PM, Matt K <ma...@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'm trying to get a custom filter to work on HBase 0.96. After some
> > > > searching, I found that starting from 0.96, the implementer is
> required
> > > to
> > > > implement "toByteArray" and "parseFrom" methods, using Protocol
> > Buffers.
> > > > But I'm having trouble with the "how".
> > > >
> > > > The "proto" file for the existing filters is located here:
> > > >
> > > >
> > >
> >
> https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/protobuf/Filter.proto
> > > >
> > > > Am I supposed to modify that file? Or create a new file, compile it
> > into
> > > > Java, and package it up with the filter?
> > > >
> > > > In the meantime, I've taken a shortcut that's not working. Here's my
> > > code:
> > > > http://pastebin.com/iHFKu9Xz
> > > >
> > > > I'm using "PrefixFilter", which comes with HBase, since I'm also
> > > filtering
> > > > by "prefix". However, that errors out with the following:
> > > > http://pastebin.com/zBg47p6Z
> > > >
> > > > Thanks in advance for helping!
> > > >
> > > > -Matt
> > > >
> > >
> >
>
Re: custom filter on hbase 0.96
Posted by Ted Yu <yu...@gmail.com>.
Matt:
I was answering Nishanth's question - testing custom filter on client side.
Cheers
On Wed, Oct 15, 2014 at 2:54 PM, Matt K <ma...@gmail.com> wrote:
> Ted, sorry, I have no idea what you mean. Could you please elaborate?
>
> On Wednesday, October 15, 2014, Ted Yu <yu...@gmail.com> wrote:
>
> > Nishanth:
> > Good question.
> >
> > As a general coding guide, writing unit test is always a good start.
> Using
> > Matt's case as an example, take a look at TestPrefixFilter.
> >
> > There're various unit tests for Filters in hbase code.
> >
> > Cheers
> >
> > On Wed, Oct 15, 2014 at 2:30 PM, Nishanth S <nishanth.2884@gmail.com
> > <javascript:;>> wrote:
> >
> > > Hi Ted ,
> > > Since I am also working on similar thing is there a way we can first
> > test
> > > the filter on client side?.You know what I mean without disrupting
> > others
> > > who are using the same cluster for other work?
> > >
> > > Thanks,
> > > Nishanth
> > >
> > > On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu <yuzhihong@gmail.com
> > <javascript:;>> wrote:
> > >
> > > > bq. Or create a new file, compile it into ...
> > > >
> > > > You should go with the above approach.
> > > >
> > > > On Wed, Oct 15, 2014 at 2:08 PM, Matt K <matvey1414@gmail.com
> > <javascript:;>> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'm trying to get a custom filter to work on HBase 0.96. After some
> > > > > searching, I found that starting from 0.96, the implementer is
> > required
> > > > to
> > > > > implement "toByteArray" and "parseFrom" methods, using Protocol
> > > Buffers.
> > > > > But I'm having trouble with the "how".
> > > > >
> > > > > The "proto" file for the existing filters is located here:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/protobuf/Filter.proto
> > > > >
> > > > > Am I supposed to modify that file? Or create a new file, compile it
> > > into
> > > > > Java, and package it up with the filter?
> > > > >
> > > > > In the meantime, I've taken a shortcut that's not working. Here's
> my
> > > > code:
> > > > > http://pastebin.com/iHFKu9Xz
> > > > >
> > > > > I'm using "PrefixFilter", which comes with HBase, since I'm also
> > > > filtering
> > > > > by "prefix". However, that errors out with the following:
> > > > > http://pastebin.com/zBg47p6Z
> > > > >
> > > > > Thanks in advance for helping!
> > > > >
> > > > > -Matt
> > > > >
> > > >
> > >
> >
>
>
> --
> www.calcmachine.com - easy online calculator.
>
Re: custom filter on hbase 0.96
Posted by Matt K <ma...@gmail.com>.
Ted, sorry, I have no idea what you mean. Could you please elaborate?
On Wednesday, October 15, 2014, Ted Yu <yu...@gmail.com> wrote:
> Nishanth:
> Good question.
>
> As a general coding guide, writing unit test is always a good start. Using
> Matt's case as an example, take a look at TestPrefixFilter.
>
> There're various unit tests for Filters in hbase code.
>
> Cheers
>
> On Wed, Oct 15, 2014 at 2:30 PM, Nishanth S <nishanth.2884@gmail.com
> <javascript:;>> wrote:
>
> > Hi Ted ,
> > Since I am also working on similar thing is there a way we can first
> test
> > the filter on client side?.You know what I mean without disrupting
> others
> > who are using the same cluster for other work?
> >
> > Thanks,
> > Nishanth
> >
> > On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu <yuzhihong@gmail.com
> <javascript:;>> wrote:
> >
> > > bq. Or create a new file, compile it into ...
> > >
> > > You should go with the above approach.
> > >
> > > On Wed, Oct 15, 2014 at 2:08 PM, Matt K <matvey1414@gmail.com
> <javascript:;>> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'm trying to get a custom filter to work on HBase 0.96. After some
> > > > searching, I found that starting from 0.96, the implementer is
> required
> > > to
> > > > implement "toByteArray" and "parseFrom" methods, using Protocol
> > Buffers.
> > > > But I'm having trouble with the "how".
> > > >
> > > > The "proto" file for the existing filters is located here:
> > > >
> > > >
> > >
> >
> https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/protobuf/Filter.proto
> > > >
> > > > Am I supposed to modify that file? Or create a new file, compile it
> > into
> > > > Java, and package it up with the filter?
> > > >
> > > > In the meantime, I've taken a shortcut that's not working. Here's my
> > > code:
> > > > http://pastebin.com/iHFKu9Xz
> > > >
> > > > I'm using "PrefixFilter", which comes with HBase, since I'm also
> > > filtering
> > > > by "prefix". However, that errors out with the following:
> > > > http://pastebin.com/zBg47p6Z
> > > >
> > > > Thanks in advance for helping!
> > > >
> > > > -Matt
> > > >
> > >
> >
>
--
www.calcmachine.com - easy online calculator.
Re: custom filter on hbase 0.96
Posted by Ted Yu <yu...@gmail.com>.
Nishanth:
Good question.
As a general coding guide, writing unit test is always a good start. Using
Matt's case as an example, take a look at TestPrefixFilter.
There're various unit tests for Filters in hbase code.
Cheers
On Wed, Oct 15, 2014 at 2:30 PM, Nishanth S <ni...@gmail.com> wrote:
> Hi Ted ,
> Since I am also working on similar thing is there a way we can first test
> the filter on client side?.You know what I mean without disrupting others
> who are using the same cluster for other work?
>
> Thanks,
> Nishanth
>
> On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > bq. Or create a new file, compile it into ...
> >
> > You should go with the above approach.
> >
> > On Wed, Oct 15, 2014 at 2:08 PM, Matt K <ma...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > I'm trying to get a custom filter to work on HBase 0.96. After some
> > > searching, I found that starting from 0.96, the implementer is required
> > to
> > > implement "toByteArray" and "parseFrom" methods, using Protocol
> Buffers.
> > > But I'm having trouble with the "how".
> > >
> > > The "proto" file for the existing filters is located here:
> > >
> > >
> >
> https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/protobuf/Filter.proto
> > >
> > > Am I supposed to modify that file? Or create a new file, compile it
> into
> > > Java, and package it up with the filter?
> > >
> > > In the meantime, I've taken a shortcut that's not working. Here's my
> > code:
> > > http://pastebin.com/iHFKu9Xz
> > >
> > > I'm using "PrefixFilter", which comes with HBase, since I'm also
> > filtering
> > > by "prefix". However, that errors out with the following:
> > > http://pastebin.com/zBg47p6Z
> > >
> > > Thanks in advance for helping!
> > >
> > > -Matt
> > >
> >
>
Re: custom filter on hbase 0.96
Posted by Nishanth S <ni...@gmail.com>.
Hi Ted ,
Since I am also working on similar thing is there a way we can first test
the filter on client side?.You know what I mean without disrupting others
who are using the same cluster for other work?
Thanks,
Nishanth
On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu <yu...@gmail.com> wrote:
> bq. Or create a new file, compile it into ...
>
> You should go with the above approach.
>
> On Wed, Oct 15, 2014 at 2:08 PM, Matt K <ma...@gmail.com> wrote:
>
> > Hi all,
> >
> > I'm trying to get a custom filter to work on HBase 0.96. After some
> > searching, I found that starting from 0.96, the implementer is required
> to
> > implement "toByteArray" and "parseFrom" methods, using Protocol Buffers.
> > But I'm having trouble with the "how".
> >
> > The "proto" file for the existing filters is located here:
> >
> >
> https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/protobuf/Filter.proto
> >
> > Am I supposed to modify that file? Or create a new file, compile it into
> > Java, and package it up with the filter?
> >
> > In the meantime, I've taken a shortcut that's not working. Here's my
> code:
> > http://pastebin.com/iHFKu9Xz
> >
> > I'm using "PrefixFilter", which comes with HBase, since I'm also
> filtering
> > by "prefix". However, that errors out with the following:
> > http://pastebin.com/zBg47p6Z
> >
> > Thanks in advance for helping!
> >
> > -Matt
> >
>
Re: custom filter on hbase 0.96
Posted by Ted Yu <yu...@gmail.com>.
bq. Or create a new file, compile it into ...
You should go with the above approach.
On Wed, Oct 15, 2014 at 2:08 PM, Matt K <ma...@gmail.com> wrote:
> Hi all,
>
> I'm trying to get a custom filter to work on HBase 0.96. After some
> searching, I found that starting from 0.96, the implementer is required to
> implement "toByteArray" and "parseFrom" methods, using Protocol Buffers.
> But I'm having trouble with the "how".
>
> The "proto" file for the existing filters is located here:
>
> https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/protobuf/Filter.proto
>
> Am I supposed to modify that file? Or create a new file, compile it into
> Java, and package it up with the filter?
>
> In the meantime, I've taken a shortcut that's not working. Here's my code:
> http://pastebin.com/iHFKu9Xz
>
> I'm using "PrefixFilter", which comes with HBase, since I'm also filtering
> by "prefix". However, that errors out with the following:
> http://pastebin.com/zBg47p6Z
>
> Thanks in advance for helping!
>
> -Matt
>