You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Sriram Muthuswamy Chittathoor <sr...@ivycomptech.com> on 2010/01/13 18:28:04 UTC
Data processing/filtering on the server
Hi:
I am currently using the HBase client api to fetch the data rows and
process the rows on the client JVM. Is there a way to specify filters
which will apply say on the region servers so that the data that is
returned to me is a filtered set. I don't want all the data rows to
travel on the wire from the server to the client JVM
Sriram C
This email is sent for and on behalf of Ivy Comptech Private Limited. Ivy Comptech Private Limited is a limited liability company.
This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system.
Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract on behalf of Ivy Comptech Private Limited unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk.
Registered office:
Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara Hills, Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994. Registered in India. A list of members' names is available for inspection at the registered office.
Re: Data processing/filtering on the server
Posted by Gary Helmling <gh...@gmail.com>.
On Thu, Jan 14, 2010 at 12:25 PM, Edward Capriolo <ed...@gmail.com>wrote:
> It does not seem like http://www.zeroturnaround.com/liverebel/, is
> available yet. Also this is a request form for an open source license,
> but no copy of the license (that I can find). Thus it is hard to
> determine if an apache product can use this.
>
>
Sorry, I didn't mean to imply this as a solution that should be adopted or
integrated into HBase. Just that if a particular user had this need for
their installation, they could make use of this tool in their setup.
I've written a couple custom filters. But for me the need does not come up
frequently.
--gh
Re: Data processing/filtering on the server
Posted by Edward Capriolo <ed...@gmail.com>.
It does not seem like http://www.zeroturnaround.com/liverebel/, is
available yet. Also this is a request form for an open source license,
but no copy of the license (that I can find). Thus it is hard to
determine if an apache product can use this.
As this discussion pertains to filters, how many custom filters would
people be making and how often would they need to be deployed?
I can see that the current filter set is light, but I could not
envision the normal use case would every require such rapid filter
development? Does anyone have a need to say make and deploy a new
filter each day? Having a very complex filter loading infrastructure,
when most people get along with the standard filters and may only need
to deploy a new filter periodically.
On Thu, Jan 14, 2010 at 11:26 AM, Gary Helmling <gh...@gmail.com> wrote:
> You may actually be able to use JRebel (
> http://www.zeroturnaround.com/jrebel/) or alternately their new LiveRebel
> "production" version (http://www.zeroturnaround.com/liverebel/) for this.
> It does hot loading of updated classes.
>
> I have not used it with HBase myself though, so no promises.
>
> --gh
>
>
> On Thu, Jan 14, 2010 at 11:14 AM, Edward Capriolo <ed...@gmail.com>wrote:
>
>> Hive lets you hot-deply UDF/jar, but I think that is a more simple.
>> Each hive query is really its own map reduce job. HBASE is a running
>> process.
>>
>> http://www.javalobby.org/java/forums/t18345.html
>>
>> Hot loading java classes does work, but you have to design interfaces
>> to deal with compile/time run time issues. Also Hot loading, has a big
>> problem hot unloading! In my experience once you get a class 'hot
>> loaded' hot unloading can not happen until you lose any references to
>> objects of the class.
>>
>> I can imagine a situation where you load a filter with a small logic
>> error, getting it unloaded and a new version loaded can be quite
>> tricky. So hot loading is cool but on the flip side it involves
>> writing a lot of code you did not need before.
>>
>> On Thu, Jan 14, 2010 at 11:00 AM, Leen Toelen <to...@gmail.com> wrote:
>> > Hi,
>> >
>> > is HBase able to add/load a jar to its classpath on the fly? It would be
>> > neat to have a 'hot deploy' directory, much like felix fileinstall.
>> >
>> > Regards,
>> > Leen
>> >
>> > On Thu, Jan 14, 2010 at 4:00 PM, Paul Ambrose <pa...@mac.com> wrote:
>> >
>> >> Yes, add it to HBASE_CLASSPATH in $HBASE_HOME/conf/hbase-env.sh
>> >> http://www.hbql.com/installation.html
>> >>
>> >> On Jan 14, 2010, at 1:39 AM, Sriram Muthuswamy Chittathoor wrote:
>> >>
>> >> > Could this be the reason as to why in HBQL something like this does
>> not
>> >> > work
>> >> > " SELECT * FROM demo1 WITH SERVER FILTER WHERE f1:val2 > 100"
>> >> >
>> >> > Whereas "SELECT * FROM demo1 WITH CLIENT FILTER WHERE f1:val2 > 100"
>> >> > works.
>> >> >
>> >> > I need to copy hbql jar to all the machines and set it in the class
>> path
>> >> > ?
>> >> >
>> >> > -----Original Message-----
>> >> > From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
>> >> > stack
>> >> > Sent: Wednesday, January 13, 2010 11:42 PM
>> >> > To: hbase-user@hadoop.apache.org
>> >> > Subject: Re: Data processing/filtering on the server
>> >> >
>> >> > Your custom code needs to be on the CLASSPATH of both server and
>> client
>> >> > so
>> >> > it may need a restart of your servers after adding your new filter
>> >> > serverside. We do not yet have dynamic loading of filters
>> implemented.
>> >> > There may be another issue where instead of passing class names, we
>> >> > pass a
>> >> > code -- saves on message sizes passed -- and your new class may not
>> make
>> >> > it
>> >> > across because currently codes need to be added to the head of
>> >> > HbaseObjectWritable. That said there is provision for when a code
>> does
>> >> > not
>> >> > exist, we instead send over the class name. Maybe this will work for
>> >> > you.
>> >> >
>> >> > St.Ack
>> >> >
>> >> > On Wed, Jan 13, 2010 at 9:38 AM, Sriram Muthuswamy Chittathoor <
>> >> > sriramc@ivycomptech.com> wrote:
>> >> >
>> >> >> Thanks Edward. As an experiment I was trying to write my own filter
>> >> > and
>> >> >> use the scan.setFilter (..) method to set it. Is this even possible
>> >> > as I
>> >> >> got into issues -- started giving some region server exception.
>> >> > Based on
>> >> >> your response if the filtering is applied on the server side
>> obviously
>> >> > my
>> >> >> local custom filter class cannot be used. Am I guessing it right ??
>> >> >>
>> >> >> -----Original Message-----
>> >> >> From: Edward Capriolo [mailto:edlinuxguru@gmail.com]
>> >> >> Sent: Wednesday, January 13, 2010 11:00 PM
>> >> >> To: hbase-user@hadoop.apache.org
>> >> >> Subject: Re: Data processing/filtering on the server
>> >> >>
>> >> >> Filters are applied server side:
>> >> >>
>> >> >>
>> >> >>
>> >> >
>> http://hadoop.apache.org/hbase/docs/current/api/org/apache/hadoop/hbase/
>> >> > filter/package-summary.html
>> >> >>
>> >> >> Edward
>> >> >> On Wed, Jan 13, 2010 at 12:28 PM, Sriram Muthuswamy Chittathoor
>> >> >> <sr...@ivycomptech.com> wrote:
>> >> >>> Hi:
>> >> >>>
>> >> >>> I am currently using the HBase client api to fetch the data rows and
>> >> >>> process the rows on the client JVM. Is there a way to specify
>> >> > filters
>> >> >>> which will apply say on the region servers so that the data that is
>> >> >>> returned to me is a filtered set. I don't want all the data rows to
>> >> >>> travel on the wire from the server to the client JVM
>> >> >>>
>> >> >>> Sriram C
>> >> >>>
>> >> >>> This email is sent for and on behalf of Ivy Comptech Private
>> >> > Limited. Ivy
>> >> >> Comptech Private Limited is a limited liability company.
>> >> >>>
>> >> >>> This email and any attachments are confidential, and may be legally
>> >> >> privileged and protected by copyright. If you are not the intended
>> >> > recipient
>> >> >> dissemination or copying of this email is prohibited. If you have
>> >> > received
>> >> >> this in error, please notify the sender by replying by email and then
>> >> > delete
>> >> >> the email completely from your system.
>> >> >>> Any views or opinions are solely those of the sender. This
>> >> > communication
>> >> >> is not intended to form a binding contract on behalf of Ivy Comptech
>> >> > Private
>> >> >> Limited unless expressly indicated to the contrary and properly
>> >> > authorised.
>> >> >> Any actions taken on the basis of this email are at the recipient's
>> >> > own
>> >> >> risk.
>> >> >>>
>> >> >>> Registered office:
>> >> >>> Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara
>> >> > Hills,
>> >> >> Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994.
>> >> >> Registered in India. A list of members' names is available for
>> >> > inspection at
>> >> >> the registered office.
>> >> >>>
>> >> >>>
>> >> >>
>> >> >> This email is sent for and on behalf of Ivy Comptech Private Limited.
>> >> > Ivy
>> >> >> Comptech Private Limited is a limited liability company.
>> >> >>
>> >> >> This email and any attachments are confidential, and may be legally
>> >> >> privileged and protected by copyright. If you are not the intended
>> >> > recipient
>> >> >> dissemination or copying of this email is prohibited. If you have
>> >> > received
>> >> >> this in error, please notify the sender by replying by email and then
>> >> > delete
>> >> >> the email completely from your system.
>> >> >> Any views or opinions are solely those of the sender. This
>> >> > communication
>> >> >> is not intended to form a binding contract on behalf of Ivy Comptech
>> >> > Private
>> >> >> Limited unless expressly indicated to the contrary and properly
>> >> > authorised.
>> >> >> Any actions taken on the basis of this email are at the recipient's
>> >> > own
>> >> >> risk.
>> >> >>
>> >> >> Registered office:
>> >> >> Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara
>> Hills,
>> >> >> Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994.
>> >> >> Registered in India. A list of members' names is available for
>> >> > inspection at
>> >> >> the registered office.
>> >> >>
>> >> >>
>> >>
>> >>
>> >
>>
>
Re: Data processing/filtering on the server
Posted by Andrew Purtell <ap...@apache.org>.
Not yet in any released version but there is a patch available if you
want to try it out:
http://issues.apache.org/jira/browse/HBASE-1936
- Andy
On Thu, Jan 14, 2010 at 11:00 AM, Leen Toelen wrote:
> Hi,
>
> is HBase able to add/load a jar to its classpath on the fly? It would be
> neat to have a 'hot deploy' directory, much like felix fileinstall.
>
> Regards,
> Leen
Re: Data processing/filtering on the server
Posted by Gary Helmling <gh...@gmail.com>.
You may actually be able to use JRebel (
http://www.zeroturnaround.com/jrebel/) or alternately their new LiveRebel
"production" version (http://www.zeroturnaround.com/liverebel/) for this.
It does hot loading of updated classes.
I have not used it with HBase myself though, so no promises.
--gh
On Thu, Jan 14, 2010 at 11:14 AM, Edward Capriolo <ed...@gmail.com>wrote:
> Hive lets you hot-deply UDF/jar, but I think that is a more simple.
> Each hive query is really its own map reduce job. HBASE is a running
> process.
>
> http://www.javalobby.org/java/forums/t18345.html
>
> Hot loading java classes does work, but you have to design interfaces
> to deal with compile/time run time issues. Also Hot loading, has a big
> problem hot unloading! In my experience once you get a class 'hot
> loaded' hot unloading can not happen until you lose any references to
> objects of the class.
>
> I can imagine a situation where you load a filter with a small logic
> error, getting it unloaded and a new version loaded can be quite
> tricky. So hot loading is cool but on the flip side it involves
> writing a lot of code you did not need before.
>
> On Thu, Jan 14, 2010 at 11:00 AM, Leen Toelen <to...@gmail.com> wrote:
> > Hi,
> >
> > is HBase able to add/load a jar to its classpath on the fly? It would be
> > neat to have a 'hot deploy' directory, much like felix fileinstall.
> >
> > Regards,
> > Leen
> >
> > On Thu, Jan 14, 2010 at 4:00 PM, Paul Ambrose <pa...@mac.com> wrote:
> >
> >> Yes, add it to HBASE_CLASSPATH in $HBASE_HOME/conf/hbase-env.sh
> >> http://www.hbql.com/installation.html
> >>
> >> On Jan 14, 2010, at 1:39 AM, Sriram Muthuswamy Chittathoor wrote:
> >>
> >> > Could this be the reason as to why in HBQL something like this does
> not
> >> > work
> >> > " SELECT * FROM demo1 WITH SERVER FILTER WHERE f1:val2 > 100"
> >> >
> >> > Whereas "SELECT * FROM demo1 WITH CLIENT FILTER WHERE f1:val2 > 100"
> >> > works.
> >> >
> >> > I need to copy hbql jar to all the machines and set it in the class
> path
> >> > ?
> >> >
> >> > -----Original Message-----
> >> > From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
> >> > stack
> >> > Sent: Wednesday, January 13, 2010 11:42 PM
> >> > To: hbase-user@hadoop.apache.org
> >> > Subject: Re: Data processing/filtering on the server
> >> >
> >> > Your custom code needs to be on the CLASSPATH of both server and
> client
> >> > so
> >> > it may need a restart of your servers after adding your new filter
> >> > serverside. We do not yet have dynamic loading of filters
> implemented.
> >> > There may be another issue where instead of passing class names, we
> >> > pass a
> >> > code -- saves on message sizes passed -- and your new class may not
> make
> >> > it
> >> > across because currently codes need to be added to the head of
> >> > HbaseObjectWritable. That said there is provision for when a code
> does
> >> > not
> >> > exist, we instead send over the class name. Maybe this will work for
> >> > you.
> >> >
> >> > St.Ack
> >> >
> >> > On Wed, Jan 13, 2010 at 9:38 AM, Sriram Muthuswamy Chittathoor <
> >> > sriramc@ivycomptech.com> wrote:
> >> >
> >> >> Thanks Edward. As an experiment I was trying to write my own filter
> >> > and
> >> >> use the scan.setFilter (..) method to set it. Is this even possible
> >> > as I
> >> >> got into issues -- started giving some region server exception.
> >> > Based on
> >> >> your response if the filtering is applied on the server side
> obviously
> >> > my
> >> >> local custom filter class cannot be used. Am I guessing it right ??
> >> >>
> >> >> -----Original Message-----
> >> >> From: Edward Capriolo [mailto:edlinuxguru@gmail.com]
> >> >> Sent: Wednesday, January 13, 2010 11:00 PM
> >> >> To: hbase-user@hadoop.apache.org
> >> >> Subject: Re: Data processing/filtering on the server
> >> >>
> >> >> Filters are applied server side:
> >> >>
> >> >>
> >> >>
> >> >
> http://hadoop.apache.org/hbase/docs/current/api/org/apache/hadoop/hbase/
> >> > filter/package-summary.html
> >> >>
> >> >> Edward
> >> >> On Wed, Jan 13, 2010 at 12:28 PM, Sriram Muthuswamy Chittathoor
> >> >> <sr...@ivycomptech.com> wrote:
> >> >>> Hi:
> >> >>>
> >> >>> I am currently using the HBase client api to fetch the data rows and
> >> >>> process the rows on the client JVM. Is there a way to specify
> >> > filters
> >> >>> which will apply say on the region servers so that the data that is
> >> >>> returned to me is a filtered set. I don't want all the data rows to
> >> >>> travel on the wire from the server to the client JVM
> >> >>>
> >> >>> Sriram C
> >> >>>
> >> >>> This email is sent for and on behalf of Ivy Comptech Private
> >> > Limited. Ivy
> >> >> Comptech Private Limited is a limited liability company.
> >> >>>
> >> >>> This email and any attachments are confidential, and may be legally
> >> >> privileged and protected by copyright. If you are not the intended
> >> > recipient
> >> >> dissemination or copying of this email is prohibited. If you have
> >> > received
> >> >> this in error, please notify the sender by replying by email and then
> >> > delete
> >> >> the email completely from your system.
> >> >>> Any views or opinions are solely those of the sender. This
> >> > communication
> >> >> is not intended to form a binding contract on behalf of Ivy Comptech
> >> > Private
> >> >> Limited unless expressly indicated to the contrary and properly
> >> > authorised.
> >> >> Any actions taken on the basis of this email are at the recipient's
> >> > own
> >> >> risk.
> >> >>>
> >> >>> Registered office:
> >> >>> Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara
> >> > Hills,
> >> >> Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994.
> >> >> Registered in India. A list of members' names is available for
> >> > inspection at
> >> >> the registered office.
> >> >>>
> >> >>>
> >> >>
> >> >> This email is sent for and on behalf of Ivy Comptech Private Limited.
> >> > Ivy
> >> >> Comptech Private Limited is a limited liability company.
> >> >>
> >> >> This email and any attachments are confidential, and may be legally
> >> >> privileged and protected by copyright. If you are not the intended
> >> > recipient
> >> >> dissemination or copying of this email is prohibited. If you have
> >> > received
> >> >> this in error, please notify the sender by replying by email and then
> >> > delete
> >> >> the email completely from your system.
> >> >> Any views or opinions are solely those of the sender. This
> >> > communication
> >> >> is not intended to form a binding contract on behalf of Ivy Comptech
> >> > Private
> >> >> Limited unless expressly indicated to the contrary and properly
> >> > authorised.
> >> >> Any actions taken on the basis of this email are at the recipient's
> >> > own
> >> >> risk.
> >> >>
> >> >> Registered office:
> >> >> Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara
> Hills,
> >> >> Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994.
> >> >> Registered in India. A list of members' names is available for
> >> > inspection at
> >> >> the registered office.
> >> >>
> >> >>
> >>
> >>
> >
>
Re: Data processing/filtering on the server
Posted by Leen Toelen <to...@gmail.com>.
OSGI classloading can unload classes as well. It can even run multiple
versions of the same class in the same vm. But I guess HBase is a bit tricky
since the classes need to be loaded in multiple vm's, right?
Leen
On Thu, Jan 14, 2010 at 5:14 PM, Edward Capriolo <ed...@gmail.com>wrote:
> Hive lets you hot-deply UDF/jar, but I think that is a more simple.
> Each hive query is really its own map reduce job. HBASE is a running
> process.
>
> http://www.javalobby.org/java/forums/t18345.html
>
> Hot loading java classes does work, but you have to design interfaces
> to deal with compile/time run time issues. Also Hot loading, has a big
> problem hot unloading! In my experience once you get a class 'hot
> loaded' hot unloading can not happen until you lose any references to
> objects of the class.
>
> I can imagine a situation where you load a filter with a small logic
> error, getting it unloaded and a new version loaded can be quite
> tricky. So hot loading is cool but on the flip side it involves
> writing a lot of code you did not need before.
>
> On Thu, Jan 14, 2010 at 11:00 AM, Leen Toelen <to...@gmail.com> wrote:
> > Hi,
> >
> > is HBase able to add/load a jar to its classpath on the fly? It would be
> > neat to have a 'hot deploy' directory, much like felix fileinstall.
> >
> > Regards,
> > Leen
> >
> > On Thu, Jan 14, 2010 at 4:00 PM, Paul Ambrose <pa...@mac.com> wrote:
> >
> >> Yes, add it to HBASE_CLASSPATH in $HBASE_HOME/conf/hbase-env.sh
> >> http://www.hbql.com/installation.html
> >>
> >> On Jan 14, 2010, at 1:39 AM, Sriram Muthuswamy Chittathoor wrote:
> >>
> >> > Could this be the reason as to why in HBQL something like this does
> not
> >> > work
> >> > " SELECT * FROM demo1 WITH SERVER FILTER WHERE f1:val2 > 100"
> >> >
> >> > Whereas "SELECT * FROM demo1 WITH CLIENT FILTER WHERE f1:val2 > 100"
> >> > works.
> >> >
> >> > I need to copy hbql jar to all the machines and set it in the class
> path
> >> > ?
> >> >
> >> > -----Original Message-----
> >> > From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
> >> > stack
> >> > Sent: Wednesday, January 13, 2010 11:42 PM
> >> > To: hbase-user@hadoop.apache.org
> >> > Subject: Re: Data processing/filtering on the server
> >> >
> >> > Your custom code needs to be on the CLASSPATH of both server and
> client
> >> > so
> >> > it may need a restart of your servers after adding your new filter
> >> > serverside. We do not yet have dynamic loading of filters
> implemented.
> >> > There may be another issue where instead of passing class names, we
> >> > pass a
> >> > code -- saves on message sizes passed -- and your new class may not
> make
> >> > it
> >> > across because currently codes need to be added to the head of
> >> > HbaseObjectWritable. That said there is provision for when a code
> does
> >> > not
> >> > exist, we instead send over the class name. Maybe this will work for
> >> > you.
> >> >
> >> > St.Ack
> >> >
> >> > On Wed, Jan 13, 2010 at 9:38 AM, Sriram Muthuswamy Chittathoor <
> >> > sriramc@ivycomptech.com> wrote:
> >> >
> >> >> Thanks Edward. As an experiment I was trying to write my own filter
> >> > and
> >> >> use the scan.setFilter (..) method to set it. Is this even possible
> >> > as I
> >> >> got into issues -- started giving some region server exception.
> >> > Based on
> >> >> your response if the filtering is applied on the server side
> obviously
> >> > my
> >> >> local custom filter class cannot be used. Am I guessing it right ??
> >> >>
> >> >> -----Original Message-----
> >> >> From: Edward Capriolo [mailto:edlinuxguru@gmail.com]
> >> >> Sent: Wednesday, January 13, 2010 11:00 PM
> >> >> To: hbase-user@hadoop.apache.org
> >> >> Subject: Re: Data processing/filtering on the server
> >> >>
> >> >> Filters are applied server side:
> >> >>
> >> >>
> >> >>
> >> >
> http://hadoop.apache.org/hbase/docs/current/api/org/apache/hadoop/hbase/
> >> > filter/package-summary.html
> >> >>
> >> >> Edward
> >> >> On Wed, Jan 13, 2010 at 12:28 PM, Sriram Muthuswamy Chittathoor
> >> >> <sr...@ivycomptech.com> wrote:
> >> >>> Hi:
> >> >>>
> >> >>> I am currently using the HBase client api to fetch the data rows and
> >> >>> process the rows on the client JVM. Is there a way to specify
> >> > filters
> >> >>> which will apply say on the region servers so that the data that is
> >> >>> returned to me is a filtered set. I don't want all the data rows to
> >> >>> travel on the wire from the server to the client JVM
> >> >>>
> >> >>> Sriram C
> >> >>>
> >> >>> This email is sent for and on behalf of Ivy Comptech Private
> >> > Limited. Ivy
> >> >> Comptech Private Limited is a limited liability company.
> >> >>>
> >> >>> This email and any attachments are confidential, and may be legally
> >> >> privileged and protected by copyright. If you are not the intended
> >> > recipient
> >> >> dissemination or copying of this email is prohibited. If you have
> >> > received
> >> >> this in error, please notify the sender by replying by email and then
> >> > delete
> >> >> the email completely from your system.
> >> >>> Any views or opinions are solely those of the sender. This
> >> > communication
> >> >> is not intended to form a binding contract on behalf of Ivy Comptech
> >> > Private
> >> >> Limited unless expressly indicated to the contrary and properly
> >> > authorised.
> >> >> Any actions taken on the basis of this email are at the recipient's
> >> > own
> >> >> risk.
> >> >>>
> >> >>> Registered office:
> >> >>> Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara
> >> > Hills,
> >> >> Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994.
> >> >> Registered in India. A list of members' names is available for
> >> > inspection at
> >> >> the registered office.
> >> >>>
> >> >>>
> >> >>
> >> >> This email is sent for and on behalf of Ivy Comptech Private Limited.
> >> > Ivy
> >> >> Comptech Private Limited is a limited liability company.
> >> >>
> >> >> This email and any attachments are confidential, and may be legally
> >> >> privileged and protected by copyright. If you are not the intended
> >> > recipient
> >> >> dissemination or copying of this email is prohibited. If you have
> >> > received
> >> >> this in error, please notify the sender by replying by email and then
> >> > delete
> >> >> the email completely from your system.
> >> >> Any views or opinions are solely those of the sender. This
> >> > communication
> >> >> is not intended to form a binding contract on behalf of Ivy Comptech
> >> > Private
> >> >> Limited unless expressly indicated to the contrary and properly
> >> > authorised.
> >> >> Any actions taken on the basis of this email are at the recipient's
> >> > own
> >> >> risk.
> >> >>
> >> >> Registered office:
> >> >> Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara
> Hills,
> >> >> Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994.
> >> >> Registered in India. A list of members' names is available for
> >> > inspection at
> >> >> the registered office.
> >> >>
> >> >>
> >>
> >>
> >
>
Re: Data processing/filtering on the server
Posted by Edward Capriolo <ed...@gmail.com>.
Hive lets you hot-deply UDF/jar, but I think that is a more simple.
Each hive query is really its own map reduce job. HBASE is a running
process.
http://www.javalobby.org/java/forums/t18345.html
Hot loading java classes does work, but you have to design interfaces
to deal with compile/time run time issues. Also Hot loading, has a big
problem hot unloading! In my experience once you get a class 'hot
loaded' hot unloading can not happen until you lose any references to
objects of the class.
I can imagine a situation where you load a filter with a small logic
error, getting it unloaded and a new version loaded can be quite
tricky. So hot loading is cool but on the flip side it involves
writing a lot of code you did not need before.
On Thu, Jan 14, 2010 at 11:00 AM, Leen Toelen <to...@gmail.com> wrote:
> Hi,
>
> is HBase able to add/load a jar to its classpath on the fly? It would be
> neat to have a 'hot deploy' directory, much like felix fileinstall.
>
> Regards,
> Leen
>
> On Thu, Jan 14, 2010 at 4:00 PM, Paul Ambrose <pa...@mac.com> wrote:
>
>> Yes, add it to HBASE_CLASSPATH in $HBASE_HOME/conf/hbase-env.sh
>> http://www.hbql.com/installation.html
>>
>> On Jan 14, 2010, at 1:39 AM, Sriram Muthuswamy Chittathoor wrote:
>>
>> > Could this be the reason as to why in HBQL something like this does not
>> > work
>> > " SELECT * FROM demo1 WITH SERVER FILTER WHERE f1:val2 > 100"
>> >
>> > Whereas "SELECT * FROM demo1 WITH CLIENT FILTER WHERE f1:val2 > 100"
>> > works.
>> >
>> > I need to copy hbql jar to all the machines and set it in the class path
>> > ?
>> >
>> > -----Original Message-----
>> > From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
>> > stack
>> > Sent: Wednesday, January 13, 2010 11:42 PM
>> > To: hbase-user@hadoop.apache.org
>> > Subject: Re: Data processing/filtering on the server
>> >
>> > Your custom code needs to be on the CLASSPATH of both server and client
>> > so
>> > it may need a restart of your servers after adding your new filter
>> > serverside. We do not yet have dynamic loading of filters implemented.
>> > There may be another issue where instead of passing class names, we
>> > pass a
>> > code -- saves on message sizes passed -- and your new class may not make
>> > it
>> > across because currently codes need to be added to the head of
>> > HbaseObjectWritable. That said there is provision for when a code does
>> > not
>> > exist, we instead send over the class name. Maybe this will work for
>> > you.
>> >
>> > St.Ack
>> >
>> > On Wed, Jan 13, 2010 at 9:38 AM, Sriram Muthuswamy Chittathoor <
>> > sriramc@ivycomptech.com> wrote:
>> >
>> >> Thanks Edward. As an experiment I was trying to write my own filter
>> > and
>> >> use the scan.setFilter (..) method to set it. Is this even possible
>> > as I
>> >> got into issues -- started giving some region server exception.
>> > Based on
>> >> your response if the filtering is applied on the server side obviously
>> > my
>> >> local custom filter class cannot be used. Am I guessing it right ??
>> >>
>> >> -----Original Message-----
>> >> From: Edward Capriolo [mailto:edlinuxguru@gmail.com]
>> >> Sent: Wednesday, January 13, 2010 11:00 PM
>> >> To: hbase-user@hadoop.apache.org
>> >> Subject: Re: Data processing/filtering on the server
>> >>
>> >> Filters are applied server side:
>> >>
>> >>
>> >>
>> > http://hadoop.apache.org/hbase/docs/current/api/org/apache/hadoop/hbase/
>> > filter/package-summary.html
>> >>
>> >> Edward
>> >> On Wed, Jan 13, 2010 at 12:28 PM, Sriram Muthuswamy Chittathoor
>> >> <sr...@ivycomptech.com> wrote:
>> >>> Hi:
>> >>>
>> >>> I am currently using the HBase client api to fetch the data rows and
>> >>> process the rows on the client JVM. Is there a way to specify
>> > filters
>> >>> which will apply say on the region servers so that the data that is
>> >>> returned to me is a filtered set. I don't want all the data rows to
>> >>> travel on the wire from the server to the client JVM
>> >>>
>> >>> Sriram C
>> >>>
>> >>> This email is sent for and on behalf of Ivy Comptech Private
>> > Limited. Ivy
>> >> Comptech Private Limited is a limited liability company.
>> >>>
>> >>> This email and any attachments are confidential, and may be legally
>> >> privileged and protected by copyright. If you are not the intended
>> > recipient
>> >> dissemination or copying of this email is prohibited. If you have
>> > received
>> >> this in error, please notify the sender by replying by email and then
>> > delete
>> >> the email completely from your system.
>> >>> Any views or opinions are solely those of the sender. This
>> > communication
>> >> is not intended to form a binding contract on behalf of Ivy Comptech
>> > Private
>> >> Limited unless expressly indicated to the contrary and properly
>> > authorised.
>> >> Any actions taken on the basis of this email are at the recipient's
>> > own
>> >> risk.
>> >>>
>> >>> Registered office:
>> >>> Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara
>> > Hills,
>> >> Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994.
>> >> Registered in India. A list of members' names is available for
>> > inspection at
>> >> the registered office.
>> >>>
>> >>>
>> >>
>> >> This email is sent for and on behalf of Ivy Comptech Private Limited.
>> > Ivy
>> >> Comptech Private Limited is a limited liability company.
>> >>
>> >> This email and any attachments are confidential, and may be legally
>> >> privileged and protected by copyright. If you are not the intended
>> > recipient
>> >> dissemination or copying of this email is prohibited. If you have
>> > received
>> >> this in error, please notify the sender by replying by email and then
>> > delete
>> >> the email completely from your system.
>> >> Any views or opinions are solely those of the sender. This
>> > communication
>> >> is not intended to form a binding contract on behalf of Ivy Comptech
>> > Private
>> >> Limited unless expressly indicated to the contrary and properly
>> > authorised.
>> >> Any actions taken on the basis of this email are at the recipient's
>> > own
>> >> risk.
>> >>
>> >> Registered office:
>> >> Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara Hills,
>> >> Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994.
>> >> Registered in India. A list of members' names is available for
>> > inspection at
>> >> the registered office.
>> >>
>> >>
>>
>>
>
Re: Data processing/filtering on the server
Posted by Leen Toelen <to...@gmail.com>.
Hi,
is HBase able to add/load a jar to its classpath on the fly? It would be
neat to have a 'hot deploy' directory, much like felix fileinstall.
Regards,
Leen
On Thu, Jan 14, 2010 at 4:00 PM, Paul Ambrose <pa...@mac.com> wrote:
> Yes, add it to HBASE_CLASSPATH in $HBASE_HOME/conf/hbase-env.sh
> http://www.hbql.com/installation.html
>
> On Jan 14, 2010, at 1:39 AM, Sriram Muthuswamy Chittathoor wrote:
>
> > Could this be the reason as to why in HBQL something like this does not
> > work
> > " SELECT * FROM demo1 WITH SERVER FILTER WHERE f1:val2 > 100"
> >
> > Whereas "SELECT * FROM demo1 WITH CLIENT FILTER WHERE f1:val2 > 100"
> > works.
> >
> > I need to copy hbql jar to all the machines and set it in the class path
> > ?
> >
> > -----Original Message-----
> > From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
> > stack
> > Sent: Wednesday, January 13, 2010 11:42 PM
> > To: hbase-user@hadoop.apache.org
> > Subject: Re: Data processing/filtering on the server
> >
> > Your custom code needs to be on the CLASSPATH of both server and client
> > so
> > it may need a restart of your servers after adding your new filter
> > serverside. We do not yet have dynamic loading of filters implemented.
> > There may be another issue where instead of passing class names, we
> > pass a
> > code -- saves on message sizes passed -- and your new class may not make
> > it
> > across because currently codes need to be added to the head of
> > HbaseObjectWritable. That said there is provision for when a code does
> > not
> > exist, we instead send over the class name. Maybe this will work for
> > you.
> >
> > St.Ack
> >
> > On Wed, Jan 13, 2010 at 9:38 AM, Sriram Muthuswamy Chittathoor <
> > sriramc@ivycomptech.com> wrote:
> >
> >> Thanks Edward. As an experiment I was trying to write my own filter
> > and
> >> use the scan.setFilter (..) method to set it. Is this even possible
> > as I
> >> got into issues -- started giving some region server exception.
> > Based on
> >> your response if the filtering is applied on the server side obviously
> > my
> >> local custom filter class cannot be used. Am I guessing it right ??
> >>
> >> -----Original Message-----
> >> From: Edward Capriolo [mailto:edlinuxguru@gmail.com]
> >> Sent: Wednesday, January 13, 2010 11:00 PM
> >> To: hbase-user@hadoop.apache.org
> >> Subject: Re: Data processing/filtering on the server
> >>
> >> Filters are applied server side:
> >>
> >>
> >>
> > http://hadoop.apache.org/hbase/docs/current/api/org/apache/hadoop/hbase/
> > filter/package-summary.html
> >>
> >> Edward
> >> On Wed, Jan 13, 2010 at 12:28 PM, Sriram Muthuswamy Chittathoor
> >> <sr...@ivycomptech.com> wrote:
> >>> Hi:
> >>>
> >>> I am currently using the HBase client api to fetch the data rows and
> >>> process the rows on the client JVM. Is there a way to specify
> > filters
> >>> which will apply say on the region servers so that the data that is
> >>> returned to me is a filtered set. I don't want all the data rows to
> >>> travel on the wire from the server to the client JVM
> >>>
> >>> Sriram C
> >>>
> >>> This email is sent for and on behalf of Ivy Comptech Private
> > Limited. Ivy
> >> Comptech Private Limited is a limited liability company.
> >>>
> >>> This email and any attachments are confidential, and may be legally
> >> privileged and protected by copyright. If you are not the intended
> > recipient
> >> dissemination or copying of this email is prohibited. If you have
> > received
> >> this in error, please notify the sender by replying by email and then
> > delete
> >> the email completely from your system.
> >>> Any views or opinions are solely those of the sender. This
> > communication
> >> is not intended to form a binding contract on behalf of Ivy Comptech
> > Private
> >> Limited unless expressly indicated to the contrary and properly
> > authorised.
> >> Any actions taken on the basis of this email are at the recipient's
> > own
> >> risk.
> >>>
> >>> Registered office:
> >>> Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara
> > Hills,
> >> Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994.
> >> Registered in India. A list of members' names is available for
> > inspection at
> >> the registered office.
> >>>
> >>>
> >>
> >> This email is sent for and on behalf of Ivy Comptech Private Limited.
> > Ivy
> >> Comptech Private Limited is a limited liability company.
> >>
> >> This email and any attachments are confidential, and may be legally
> >> privileged and protected by copyright. If you are not the intended
> > recipient
> >> dissemination or copying of this email is prohibited. If you have
> > received
> >> this in error, please notify the sender by replying by email and then
> > delete
> >> the email completely from your system.
> >> Any views or opinions are solely those of the sender. This
> > communication
> >> is not intended to form a binding contract on behalf of Ivy Comptech
> > Private
> >> Limited unless expressly indicated to the contrary and properly
> > authorised.
> >> Any actions taken on the basis of this email are at the recipient's
> > own
> >> risk.
> >>
> >> Registered office:
> >> Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara Hills,
> >> Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994.
> >> Registered in India. A list of members' names is available for
> > inspection at
> >> the registered office.
> >>
> >>
>
>
Re: Data processing/filtering on the server
Posted by Paul Ambrose <pa...@mac.com>.
Yes, add it to HBASE_CLASSPATH in $HBASE_HOME/conf/hbase-env.sh
http://www.hbql.com/installation.html
On Jan 14, 2010, at 1:39 AM, Sriram Muthuswamy Chittathoor wrote:
> Could this be the reason as to why in HBQL something like this does not
> work
> " SELECT * FROM demo1 WITH SERVER FILTER WHERE f1:val2 > 100"
>
> Whereas "SELECT * FROM demo1 WITH CLIENT FILTER WHERE f1:val2 > 100"
> works.
>
> I need to copy hbql jar to all the machines and set it in the class path
> ?
>
> -----Original Message-----
> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
> stack
> Sent: Wednesday, January 13, 2010 11:42 PM
> To: hbase-user@hadoop.apache.org
> Subject: Re: Data processing/filtering on the server
>
> Your custom code needs to be on the CLASSPATH of both server and client
> so
> it may need a restart of your servers after adding your new filter
> serverside. We do not yet have dynamic loading of filters implemented.
> There may be another issue where instead of passing class names, we
> pass a
> code -- saves on message sizes passed -- and your new class may not make
> it
> across because currently codes need to be added to the head of
> HbaseObjectWritable. That said there is provision for when a code does
> not
> exist, we instead send over the class name. Maybe this will work for
> you.
>
> St.Ack
>
> On Wed, Jan 13, 2010 at 9:38 AM, Sriram Muthuswamy Chittathoor <
> sriramc@ivycomptech.com> wrote:
>
>> Thanks Edward. As an experiment I was trying to write my own filter
> and
>> use the scan.setFilter (..) method to set it. Is this even possible
> as I
>> got into issues -- started giving some region server exception.
> Based on
>> your response if the filtering is applied on the server side obviously
> my
>> local custom filter class cannot be used. Am I guessing it right ??
>>
>> -----Original Message-----
>> From: Edward Capriolo [mailto:edlinuxguru@gmail.com]
>> Sent: Wednesday, January 13, 2010 11:00 PM
>> To: hbase-user@hadoop.apache.org
>> Subject: Re: Data processing/filtering on the server
>>
>> Filters are applied server side:
>>
>>
>>
> http://hadoop.apache.org/hbase/docs/current/api/org/apache/hadoop/hbase/
> filter/package-summary.html
>>
>> Edward
>> On Wed, Jan 13, 2010 at 12:28 PM, Sriram Muthuswamy Chittathoor
>> <sr...@ivycomptech.com> wrote:
>>> Hi:
>>>
>>> I am currently using the HBase client api to fetch the data rows and
>>> process the rows on the client JVM. Is there a way to specify
> filters
>>> which will apply say on the region servers so that the data that is
>>> returned to me is a filtered set. I don't want all the data rows to
>>> travel on the wire from the server to the client JVM
>>>
>>> Sriram C
>>>
>>> This email is sent for and on behalf of Ivy Comptech Private
> Limited. Ivy
>> Comptech Private Limited is a limited liability company.
>>>
>>> This email and any attachments are confidential, and may be legally
>> privileged and protected by copyright. If you are not the intended
> recipient
>> dissemination or copying of this email is prohibited. If you have
> received
>> this in error, please notify the sender by replying by email and then
> delete
>> the email completely from your system.
>>> Any views or opinions are solely those of the sender. This
> communication
>> is not intended to form a binding contract on behalf of Ivy Comptech
> Private
>> Limited unless expressly indicated to the contrary and properly
> authorised.
>> Any actions taken on the basis of this email are at the recipient's
> own
>> risk.
>>>
>>> Registered office:
>>> Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara
> Hills,
>> Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994.
>> Registered in India. A list of members' names is available for
> inspection at
>> the registered office.
>>>
>>>
>>
>> This email is sent for and on behalf of Ivy Comptech Private Limited.
> Ivy
>> Comptech Private Limited is a limited liability company.
>>
>> This email and any attachments are confidential, and may be legally
>> privileged and protected by copyright. If you are not the intended
> recipient
>> dissemination or copying of this email is prohibited. If you have
> received
>> this in error, please notify the sender by replying by email and then
> delete
>> the email completely from your system.
>> Any views or opinions are solely those of the sender. This
> communication
>> is not intended to form a binding contract on behalf of Ivy Comptech
> Private
>> Limited unless expressly indicated to the contrary and properly
> authorised.
>> Any actions taken on the basis of this email are at the recipient's
> own
>> risk.
>>
>> Registered office:
>> Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara Hills,
>> Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994.
>> Registered in India. A list of members' names is available for
> inspection at
>> the registered office.
>>
>>
RE: Data processing/filtering on the server
Posted by Sriram Muthuswamy Chittathoor <sr...@ivycomptech.com>.
Could this be the reason as to why in HBQL something like this does not
work
" SELECT * FROM demo1 WITH SERVER FILTER WHERE f1:val2 > 100"
Whereas "SELECT * FROM demo1 WITH CLIENT FILTER WHERE f1:val2 > 100"
works.
I need to copy hbql jar to all the machines and set it in the class path
?
-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
stack
Sent: Wednesday, January 13, 2010 11:42 PM
To: hbase-user@hadoop.apache.org
Subject: Re: Data processing/filtering on the server
Your custom code needs to be on the CLASSPATH of both server and client
so
it may need a restart of your servers after adding your new filter
serverside. We do not yet have dynamic loading of filters implemented.
There may be another issue where instead of passing class names, we
pass a
code -- saves on message sizes passed -- and your new class may not make
it
across because currently codes need to be added to the head of
HbaseObjectWritable. That said there is provision for when a code does
not
exist, we instead send over the class name. Maybe this will work for
you.
St.Ack
On Wed, Jan 13, 2010 at 9:38 AM, Sriram Muthuswamy Chittathoor <
sriramc@ivycomptech.com> wrote:
> Thanks Edward. As an experiment I was trying to write my own filter
and
> use the scan.setFilter (..) method to set it. Is this even possible
as I
> got into issues -- started giving some region server exception.
Based on
> your response if the filtering is applied on the server side obviously
my
> local custom filter class cannot be used. Am I guessing it right ??
>
> -----Original Message-----
> From: Edward Capriolo [mailto:edlinuxguru@gmail.com]
> Sent: Wednesday, January 13, 2010 11:00 PM
> To: hbase-user@hadoop.apache.org
> Subject: Re: Data processing/filtering on the server
>
> Filters are applied server side:
>
>
>
http://hadoop.apache.org/hbase/docs/current/api/org/apache/hadoop/hbase/
filter/package-summary.html
>
> Edward
> On Wed, Jan 13, 2010 at 12:28 PM, Sriram Muthuswamy Chittathoor
> <sr...@ivycomptech.com> wrote:
> > Hi:
> >
> > I am currently using the HBase client api to fetch the data rows and
> > process the rows on the client JVM. Is there a way to specify
filters
> > which will apply say on the region servers so that the data that is
> > returned to me is a filtered set. I don't want all the data rows to
> > travel on the wire from the server to the client JVM
> >
> > Sriram C
> >
> > This email is sent for and on behalf of Ivy Comptech Private
Limited. Ivy
> Comptech Private Limited is a limited liability company.
> >
> > This email and any attachments are confidential, and may be legally
> privileged and protected by copyright. If you are not the intended
recipient
> dissemination or copying of this email is prohibited. If you have
received
> this in error, please notify the sender by replying by email and then
delete
> the email completely from your system.
> > Any views or opinions are solely those of the sender. This
communication
> is not intended to form a binding contract on behalf of Ivy Comptech
Private
> Limited unless expressly indicated to the contrary and properly
authorised.
> Any actions taken on the basis of this email are at the recipient's
own
> risk.
> >
> > Registered office:
> > Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara
Hills,
> Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994.
> Registered in India. A list of members' names is available for
inspection at
> the registered office.
> >
> >
>
> This email is sent for and on behalf of Ivy Comptech Private Limited.
Ivy
> Comptech Private Limited is a limited liability company.
>
> This email and any attachments are confidential, and may be legally
> privileged and protected by copyright. If you are not the intended
recipient
> dissemination or copying of this email is prohibited. If you have
received
> this in error, please notify the sender by replying by email and then
delete
> the email completely from your system.
> Any views or opinions are solely those of the sender. This
communication
> is not intended to form a binding contract on behalf of Ivy Comptech
Private
> Limited unless expressly indicated to the contrary and properly
authorised.
> Any actions taken on the basis of this email are at the recipient's
own
> risk.
>
> Registered office:
> Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara Hills,
> Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994.
> Registered in India. A list of members' names is available for
inspection at
> the registered office.
>
>
Re: Data processing/filtering on the server
Posted by stack <st...@duboce.net>.
Your custom code needs to be on the CLASSPATH of both server and client so
it may need a restart of your servers after adding your new filter
serverside. We do not yet have dynamic loading of filters implemented.
There may be another issue where instead of passing class names, we pass a
code -- saves on message sizes passed -- and your new class may not make it
across because currently codes need to be added to the head of
HbaseObjectWritable. That said there is provision for when a code does not
exist, we instead send over the class name. Maybe this will work for you.
St.Ack
On Wed, Jan 13, 2010 at 9:38 AM, Sriram Muthuswamy Chittathoor <
sriramc@ivycomptech.com> wrote:
> Thanks Edward. As an experiment I was trying to write my own filter and
> use the scan.setFilter (..) method to set it. Is this even possible as I
> got into issues -- started giving some region server exception. Based on
> your response if the filtering is applied on the server side obviously my
> local custom filter class cannot be used. Am I guessing it right ??
>
> -----Original Message-----
> From: Edward Capriolo [mailto:edlinuxguru@gmail.com]
> Sent: Wednesday, January 13, 2010 11:00 PM
> To: hbase-user@hadoop.apache.org
> Subject: Re: Data processing/filtering on the server
>
> Filters are applied server side:
>
>
> http://hadoop.apache.org/hbase/docs/current/api/org/apache/hadoop/hbase/filter/package-summary.html
>
> Edward
> On Wed, Jan 13, 2010 at 12:28 PM, Sriram Muthuswamy Chittathoor
> <sr...@ivycomptech.com> wrote:
> > Hi:
> >
> > I am currently using the HBase client api to fetch the data rows and
> > process the rows on the client JVM. Is there a way to specify filters
> > which will apply say on the region servers so that the data that is
> > returned to me is a filtered set. I don't want all the data rows to
> > travel on the wire from the server to the client JVM
> >
> > Sriram C
> >
> > This email is sent for and on behalf of Ivy Comptech Private Limited. Ivy
> Comptech Private Limited is a limited liability company.
> >
> > This email and any attachments are confidential, and may be legally
> privileged and protected by copyright. If you are not the intended recipient
> dissemination or copying of this email is prohibited. If you have received
> this in error, please notify the sender by replying by email and then delete
> the email completely from your system.
> > Any views or opinions are solely those of the sender. This communication
> is not intended to form a binding contract on behalf of Ivy Comptech Private
> Limited unless expressly indicated to the contrary and properly authorised.
> Any actions taken on the basis of this email are at the recipient's own
> risk.
> >
> > Registered office:
> > Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara Hills,
> Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994.
> Registered in India. A list of members' names is available for inspection at
> the registered office.
> >
> >
>
> This email is sent for and on behalf of Ivy Comptech Private Limited. Ivy
> Comptech Private Limited is a limited liability company.
>
> This email and any attachments are confidential, and may be legally
> privileged and protected by copyright. If you are not the intended recipient
> dissemination or copying of this email is prohibited. If you have received
> this in error, please notify the sender by replying by email and then delete
> the email completely from your system.
> Any views or opinions are solely those of the sender. This communication
> is not intended to form a binding contract on behalf of Ivy Comptech Private
> Limited unless expressly indicated to the contrary and properly authorised.
> Any actions taken on the basis of this email are at the recipient's own
> risk.
>
> Registered office:
> Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara Hills,
> Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994.
> Registered in India. A list of members' names is available for inspection at
> the registered office.
>
>
Re: Data processing/filtering on the server
Posted by Andrew Purtell <ap...@apache.org>.
Coprocessors will also let you move some forms of computation into the
region server process as that makes sense. See HBASE-2000.
http://issues.apache.org/jira/browse/HBASE-2000
This issue *is* under active development. I've not had much time as of
late but when things calm down at work I'll be back at it.
For example, with coprocessors it would be possible to run MapReduce
computations on data stored in HBase without all of the data transfer
and task scheduling overheads of involving the Hadoop MapReduce
framework. Speed ups of 100x or 1000x are not unreasonable to
contemplate.
- Andy
----- Original Message ----
> From: Sriram Muthuswamy Chittathoor <sr...@ivycomptech.com>
> To: hbase-user@hadoop.apache.org
> Sent: Wed, January 13, 2010 9:38:44 AM
> Subject: RE: Data processing/filtering on the server
>
> Thanks Edward. As an experiment I was trying to write my own filter and use the
> scan.setFilter (..) method to set it. Is this even possible as I got into
> issues -- started giving some region server exception. Based on your response
> if the filtering is applied on the server side obviously my local custom filter
> class cannot be used. Am I guessing it right ??
>
> -----Original Message-----
> From: Edward Capriolo [mailto:edlinuxguru@gmail.com]
> Sent: Wednesday, January 13, 2010 11:00 PM
> To: hbase-user@hadoop.apache.org
> Subject: Re: Data processing/filtering on the server
>
> Filters are applied server side:
>
> http://hadoop.apache.org/hbase/docs/current/api/org/apache/hadoop/hbase/filter/package-summary.html
>
> Edward
> On Wed, Jan 13, 2010 at 12:28 PM, Sriram Muthuswamy Chittathoor
> wrote:
> > Hi:
> >
> > I am currently using the HBase client api to fetch the data rows and
> > process the rows on the client JVM. Is there a way to specify filters
> > which will apply say on the region servers so that the data that is
> > returned to me is a filtered set. I don't want all the data rows to
> > travel on the wire from the server to the client JVM
> >
> > Sriram C
> >
> > This email is sent for and on behalf of Ivy Comptech Private Limited. Ivy
> Comptech Private Limited is a limited liability company.
> >
> > This email and any attachments are confidential, and may be legally privileged
> and protected by copyright. If you are not the intended recipient dissemination
> or copying of this email is prohibited. If you have received this in error,
> please notify the sender by replying by email and then delete the email
> completely from your system.
> > Any views or opinions are solely those of the sender. This communication is
> not intended to form a binding contract on behalf of Ivy Comptech Private
> Limited unless expressly indicated to the contrary and properly authorised. Any
> actions taken on the basis of this email are at the recipient's own risk.
> >
> > Registered office:
> > Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara Hills,
> Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994. Registered
> in India. A list of members' names is available for inspection at the registered
> office.
> >
> >
>
> This email is sent for and on behalf of Ivy Comptech Private Limited. Ivy
> Comptech Private Limited is a limited liability company.
>
> This email and any attachments are confidential, and may be legally privileged
> and protected by copyright. If you are not the intended recipient dissemination
> or copying of this email is prohibited. If you have received this in error,
> please notify the sender by replying by email and then delete the email
> completely from your system.
> Any views or opinions are solely those of the sender. This communication is not
> intended to form a binding contract on behalf of Ivy Comptech Private Limited
> unless expressly indicated to the contrary and properly authorised. Any actions
> taken on the basis of this email are at the recipient's own risk.
>
> Registered office:
> Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara Hills, Hyderabad
> 500 033, Andhra Pradesh, India. Registered number: 37994. Registered in India. A
> list of members' names is available for inspection at the registered office.
RE: Data processing/filtering on the server
Posted by Sriram Muthuswamy Chittathoor <sr...@ivycomptech.com>.
Thanks Edward. As an experiment I was trying to write my own filter and use the scan.setFilter (..) method to set it. Is this even possible as I got into issues -- started giving some region server exception. Based on your response if the filtering is applied on the server side obviously my local custom filter class cannot be used. Am I guessing it right ??
-----Original Message-----
From: Edward Capriolo [mailto:edlinuxguru@gmail.com]
Sent: Wednesday, January 13, 2010 11:00 PM
To: hbase-user@hadoop.apache.org
Subject: Re: Data processing/filtering on the server
Filters are applied server side:
http://hadoop.apache.org/hbase/docs/current/api/org/apache/hadoop/hbase/filter/package-summary.html
Edward
On Wed, Jan 13, 2010 at 12:28 PM, Sriram Muthuswamy Chittathoor
<sr...@ivycomptech.com> wrote:
> Hi:
>
> I am currently using the HBase client api to fetch the data rows and
> process the rows on the client JVM. Is there a way to specify filters
> which will apply say on the region servers so that the data that is
> returned to me is a filtered set. I don't want all the data rows to
> travel on the wire from the server to the client JVM
>
> Sriram C
>
> This email is sent for and on behalf of Ivy Comptech Private Limited. Ivy Comptech Private Limited is a limited liability company.
>
> This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system.
> Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract on behalf of Ivy Comptech Private Limited unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk.
>
> Registered office:
> Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara Hills, Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994. Registered in India. A list of members' names is available for inspection at the registered office.
>
>
This email is sent for and on behalf of Ivy Comptech Private Limited. Ivy Comptech Private Limited is a limited liability company.
This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system.
Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract on behalf of Ivy Comptech Private Limited unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk.
Registered office:
Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara Hills, Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994. Registered in India. A list of members' names is available for inspection at the registered office.
Re: Data processing/filtering on the server
Posted by Edward Capriolo <ed...@gmail.com>.
Filters are applied server side:
http://hadoop.apache.org/hbase/docs/current/api/org/apache/hadoop/hbase/filter/package-summary.html
Edward
On Wed, Jan 13, 2010 at 12:28 PM, Sriram Muthuswamy Chittathoor
<sr...@ivycomptech.com> wrote:
> Hi:
>
> I am currently using the HBase client api to fetch the data rows and
> process the rows on the client JVM. Is there a way to specify filters
> which will apply say on the region servers so that the data that is
> returned to me is a filtered set. I don't want all the data rows to
> travel on the wire from the server to the client JVM
>
> Sriram C
>
> This email is sent for and on behalf of Ivy Comptech Private Limited. Ivy Comptech Private Limited is a limited liability company.
>
> This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system.
> Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract on behalf of Ivy Comptech Private Limited unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk.
>
> Registered office:
> Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara Hills, Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994. Registered in India. A list of members' names is available for inspection at the registered office.
>
>