You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Marc Sturm <ma...@nyp.org> on 2015/03/18 04:04:31 UTC
RE: custom filter on hbase 0.96
Matt,
Were you able to successfully distribute your custom Filter? Yes, can you tell us how you did it?
Thanks
Marc
-----Original Message-----
From: Matt K [mailto:matvey1414@gmail.com]
Sent: Tuesday, October 21, 2014 9:02 PM
To: user@hbase.apache.org
Subject: Re: custom filter on hbase 0.96
Thanks Kevin!
I was under impression, probably mistakingly, that as of 0.96 placing the filter on hdfs under hbase lib directory is sufficient and RS should load the filter dynamically from hdfs. Is that not the case?
On Tuesday, October 21, 2014, Kevin <ke...@gmail.com> wrote:
> BTW, the error looks like you didn't distribute your custom filter to
> your region servers.
>
> On Tue, Oct 21, 2014 at 1:34 PM, Kevin <kevin.macksamie@gmail.com
> <javascript:;>> wrote:
>
> > Matt,
> >
> > You should create your own proto file and compile that with the
> > Google Protocol Buffer compiler. Take a look at the
> > SingleColumnValueFilter's
> > code:
> >
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java
> /org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java#L327
> >
> > You will need to override `public byte[] toByteArray()` and `public
> static
> > Filter parseFrom(byte[] pbBytes)`. The output of toByteArray()
> > should be the byte[] from serializing with the protocol buffer. This
> > will also be
> the
> > input to parseFrom(byte[]), which is called using reflection on the
> > server-side to instantiate your custom filter and use it.
> >
> > On Sun, Oct 19, 2014 at 11:31 AM, Matt K <matvey1414@gmail.com
> <javascript:;>> wrote:
> >
> >> Anyone?
> >>
> >> On Thursday, October 16, 2014, Matt K <matvey1414@gmail.com
> <javascript:;>> wrote:
> >>
> >> > Hi, can anyone help with above? Feels like I'm missing something
> >> obvious.
> >> >
> >> > On Wednesday, October 15, 2014, Nishanth S
> >> > <nishanth.2884@gmail.com
> <javascript:;>
> >> > <javascript:_e(%7B%7D,'cvml','nishanth.2884@gmail.com
> >> > <javascript:;>');>>
> wrote:
> >> >
> >> >> Thanks Ted .I will take a look.
> >> >>
> >> >> -Nishanth
> >> >>
> >> >> On Wed, Oct 15, 2014 at 3:43 PM, Ted Yu <yuzhihong@gmail.com
> <javascript:;>> wrote:
> >> >>
> >> >> > Nishanth:
> >> >> > Good question.
> >> >> >
> >> >> > As a general coding guide, writing unit test is always a good
> start.
> >> >> Using
> >> >> > Matt's case as an example, take a look at TestPrefixFilter.
> >> >> >
> >> >> > There're various unit tests for Filters in hbase code.
> >> >> >
> >> >> > Cheers
> >> >> >
> >> >> > On Wed, Oct 15, 2014 at 2:30 PM, Nishanth S <
> nishanth.2884@gmail.com <javascript:;>
> >> >
> >> >> > wrote:
> >> >> >
> >> >> > > Hi Ted ,
> >> >> > > Since I am also working on similar thing is there a way we
> >> >> > > can
> >> first
> >> >> > test
> >> >> > > the filter on client side?.You know what I mean without
> disrupting
> >> >> > others
> >> >> > > who are using the same cluster for other work?
> >> >> > >
> >> >> > > Thanks,
> >> >> > > Nishanth
> >> >> > >
> >> >> > > On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu <yuzhihong@gmail.com
> <javascript:;>>
> >> wrote:
> >> >> > >
> >> >> > > > bq. Or create a new file, compile it into ...
> >> >> > > >
> >> >> > > > You should go with the above approach.
> >> >> > > >
> >> >> > > > On Wed, Oct 15, 2014 at 2:08 PM, Matt K
> >> >> > > > <matvey1414@gmail.com
> <javascript:;>>
> >> >> wrote:
> >> >> > > >
> >> >> > > > > Hi all,
> >> >> > > > >
> >> >> > > > > I'm trying to get a custom filter to work on HBase 0.96.
> After
> >> >> some
> >> >> > > > > searching, I found that starting from 0.96, the
> >> >> > > > > implementer
> is
> >> >> > required
> >> >> > > > to
> >> >> > > > > implement "toByteArray" and "parseFrom" methods, using
> Protocol
> >> >> > > Buffers.
> >> >> > > > > But I'm having trouble with the "how".
> >> >> > > > >
> >> >> > > > > The "proto" file for the existing filters is located here:
> >> >> > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >>
> https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/pr
> otobuf/Filter.proto
> >> >> > > > >
> >> >> > > > > Am I supposed to modify that file? Or create a new file,
> >> compile
> >> >> it
> >> >> > > into
> >> >> > > > > Java, and package it up with the filter?
> >> >> > > > >
> >> >> > > > > In the meantime, I've taken a shortcut that's not working.
> >> Here's
> >> >> my
> >> >> > > > code:
> >> >> > > > > http://pastebin.com/iHFKu9Xz
> >> >> > > > >
> >> >> > > > > I'm using "PrefixFilter", which comes with HBase, since
> >> >> > > > > I'm
> >> also
> >> >> > > > filtering
> >> >> > > > > by "prefix". However, that errors out with the following:
> >> >> > > > > http://pastebin.com/zBg47p6Z
> >> >> > > > >
> >> >> > > > > Thanks in advance for helping!
> >> >> > > > >
> >> >> > > > > -Matt
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >> >
> >> >
> >> > --
> >> > www.calcmachine.com - easy online calculator.
> >> >
> >>
> >>
> >> --
> >> www.calcmachine.com - easy online calculator.
> >>
> >
> >
>
--
www.calcmachine.com - easy online calculator.
This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
Re: custom filter on hbase 0.96
Posted by Anil Gupta <an...@gmail.com>.
To deploy a custom filter. You just need to add the jar file of custom filter in classpath of HBase on each region server. Then do a restart of each region server.
HTH,
Anil
Sent from my iPhone
> On Apr 1, 2015, at 1:48 AM, Jeroen Hoek <je...@lable.org> wrote:
>
> I too would love to find out more about distributing custom Filter
> classes. Any pointers would be appreciated.
>
>> On 18-03-15 04:04, Marc Sturm wrote:
>> Matt, Were you able to successfully distribute your custom Filter?
>> Yes, can you tell us how you did it? Thanks Marc
>>
>>> -----Original Message----- From: Matt K
>>> [mailto:matvey1414@gmail.com] Sent: Tuesday, October 21, 2014 9:02
>>> PM To: user@hbase.apache.org Subject: Re: custom filter on hbase
>>> 0.96
>>>
>>> Thanks Kevin!
>>>
>>> I was under impression, probably mistakingly, that as of 0.96
>>> placing the filter on hdfs under hbase lib directory is sufficient
>>> and RS should load the filter dynamically from hdfs. Is that not
>>> the case?
>
> --
> Kind regards,
>
> Jeroen Hoek
> http://lable.org — KvK № 55984037 — BTW № NL8519.32.411.B.01
>
>
Re: custom filter on hbase 0.96
Posted by Jeroen Hoek <je...@lable.org>.
I too would love to find out more about distributing custom Filter
classes. Any pointers would be appreciated.
On 18-03-15 04:04, Marc Sturm wrote:
> Matt, Were you able to successfully distribute your custom Filter?
> Yes, can you tell us how you did it? Thanks Marc
>
>> -----Original Message----- From: Matt K
>> [mailto:matvey1414@gmail.com] Sent: Tuesday, October 21, 2014 9:02
>> PM To: user@hbase.apache.org Subject: Re: custom filter on hbase
>> 0.96
>>
>> Thanks Kevin!
>>
>> I was under impression, probably mistakingly, that as of 0.96
>> placing the filter on hdfs under hbase lib directory is sufficient
>> and RS should load the filter dynamically from hdfs. Is that not
>> the case?
--
Kind regards,
Jeroen Hoek
http://lable.org — KvK № 55984037 — BTW № NL8519.32.411.B.01