You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Hotec04 <ha...@dish.com> on 2014/07/09 17:54:49 UTC
How to perform fuzzyRowFilter with range search
Hi All,
I tried to find apis for fuzzyRowFilter with range search but did not find
any. Here is what I want to achieve: show the row key in a certain date
range. Our row key design is like customer id + timestamp. Example is:
1234567892014-07-09T00:11:22.333-04:00. Input should be the start date and
end date and output will be all row keys within that date period. Example:
input start date = 2014-07-07 and end date = 2014-07-10 then output will be
1234567892014-07-09T00:11:22.333-04:00 and
xxxxxxxxx2014-07-08Txx:xx:xx.xxx-04:00 which means all row keys regardless
customer id within that date range should be returned.
We are using hbase api 0.94.15-cdh4.6.0.jar and hbase is from cloudera
manager. I am not sure whether they support hbase 0.99 api.
Any help will be appreciated.
Thank you!
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/How-to-perform-fuzzyRowFilter-with-range-search-tp4061159.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: How to perform fuzzyRowFilter with range search
Posted by Hotec04 <ha...@dish.com>.
Yep...I went with the old design...loop through the date and perform
SubstringComparator for each date. The performance seems good. Thus we will
go with this solution for now.
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/How-to-perform-fuzzyRowFilter-with-range-search-tp4061159p4061191.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: How to perform fuzzyRowFilter with range search
Posted by Ted Yu <yu...@gmail.com>.
bq. [07-31]
The above would give you the following error:
Caused by: java.util.regex.PatternSyntaxException: Illegal character range
near index 4
[07-31]
^
Cheers
On Wed, Jul 9, 2014 at 10:43 AM, Hotec04 <ha...@dish.com> wrote:
> Yes, RegexStringComparator only valid with EQUAl or NOT_EQUAL, but I can
> regex given the start date and end date. It will be something like:
> "[0-9]|[0-9]|[0-9]|2014-0[7-8]-[07-31]T??:??:??.???-04:00" I still need to
> figure out how to write the regular expression in RegexStringComparator.
>
> Will this work?
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/How-to-perform-fuzzyRowFilter-with-range-search-tp4061159p4061172.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Re: How to perform fuzzyRowFilter with range search
Posted by Hotec04 <ha...@dish.com>.
Yes, RegexStringComparator only valid with EQUAl or NOT_EQUAL, but I can
regex given the start date and end date. It will be something like:
"[0-9]|[0-9]|[0-9]|2014-0[7-8]-[07-31]T??:??:??.???-04:00" I still need to
figure out how to write the regular expression in RegexStringComparator.
Will this work?
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/How-to-perform-fuzzyRowFilter-with-range-search-tp4061159p4061172.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: How to perform fuzzyRowFilter with range search
Posted by Ted Yu <yu...@gmail.com>.
See javadoc of RegexStringComparator:
* Only EQUAL or NOT_EQUAL comparisons are valid with this comparator.
Cheers
On Wed, Jul 9, 2014 at 10:29 AM, Hotec04 <ha...@dish.com> wrote:
> hmmm...just read more on RegexStringComparator...looks like this is the
> best
> option for me now. Let me play around with this api...
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/How-to-perform-fuzzyRowFilter-with-range-search-tp4061159p4061168.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Re: How to perform fuzzyRowFilter with range search
Posted by Hotec04 <ha...@dish.com>.
hmmm...just read more on RegexStringComparator...looks like this is the best
option for me now. Let me play around with this api...
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/How-to-perform-fuzzyRowFilter-with-range-search-tp4061159p4061168.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: How to perform fuzzyRowFilter with range search
Posted by Ted Yu <yu...@gmail.com>.
w.r.t. reducing length of rowkey, please see
http://hbase.apache.org/book.html#keyvalue
especially 9.7.6.6.1
the bigger the KeyValue is, the fewer KeyValue's are held in block cache.
On Wed, Jul 9, 2014 at 10:18 AM, Hotec04 <ha...@dish.com> wrote:
> SubstringComparator does not have Greater_OR_EQUAL and LESS_OR_EQUAL for
> CompareOp, I think I need that for date range (greater than the start date
> and less than the end date).
>
> Right now we are using HBase's UI, maybe in future we can create one above
> HBase's UI.
>
> Thank you!
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/How-to-perform-fuzzyRowFilter-with-range-search-tp4061159p4061167.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Re: How to perform fuzzyRowFilter with range search
Posted by Hotec04 <ha...@dish.com>.
SubstringComparator does not have Greater_OR_EQUAL and LESS_OR_EQUAL for
CompareOp, I think I need that for date range (greater than the start date
and less than the end date).
Right now we are using HBase's UI, maybe in future we can create one above
HBase's UI.
Thank you!
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/How-to-perform-fuzzyRowFilter-with-range-search-tp4061159p4061167.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: How to perform fuzzyRowFilter with range search
Posted by Ted Yu <yu...@gmail.com>.
bq. Is there a BinaryPostfixComparator
Have you looked at SubstringComparator ?
bq. but that is not user friendly as shown 1404921543369
Do you have Web UI other than HBase's UI ?
You can convert epoch time to human readable form there.
Cheers
On Wed, Jul 9, 2014 at 9:37 AM, Hotec04 <ha...@dish.com> wrote:
> I implemented a range search on rowFilter for given customer id +
> timestamp.
> That one works fine. Just come out of my mind...should I use 000000000 to
> 999999999 as customer id then plus the timestamp in rowFilter with
> CompareOp
> and BinaryComparator? so the start key will be
> 0000000002014-07-07T00:00:00.000-04:00 and end key will be
> 9999999992014-07-10T23:59:59.999-04:00. Will this resolve my issues?
>
> We thought of that way but that is not user friendly as shown 1404921543369
> in the rowkey, user does not understand what that means.
>
> Thank you!
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/How-to-perform-fuzzyRowFilter-with-range-search-tp4061159p4061162.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Re: How to perform fuzzyRowFilter with range search
Posted by Hotec04 <ha...@dish.com>.
This one does not perform well on date range...There are records out of the
date range been returned. Is there a BinaryPostfixComparator so I can go
from backward?
Thank you!
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/How-to-perform-fuzzyRowFilter-with-range-search-tp4061159p4061165.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: How to perform fuzzyRowFilter with range search
Posted by Hotec04 <ha...@dish.com>.
I implemented a range search on rowFilter for given customer id + timestamp.
That one works fine. Just come out of my mind...should I use 000000000 to
999999999 as customer id then plus the timestamp in rowFilter with CompareOp
and BinaryComparator? so the start key will be
0000000002014-07-07T00:00:00.000-04:00 and end key will be
9999999992014-07-10T23:59:59.999-04:00. Will this resolve my issues?
We thought of that way but that is not user friendly as shown 1404921543369
in the rowkey, user does not understand what that means.
Thank you!
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/How-to-perform-fuzzyRowFilter-with-range-search-tp4061159p4061162.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: How to perform fuzzyRowFilter with range search
Posted by Ted Yu <yu...@gmail.com>.
fuzzyRowFilter is not amenable to range search.
bq. 2014-07-09T00:11:22.333-04:00
Just curious: are you embedding string form of date in rowkey ?
Have you considered using epoch time ?
Cheers
On Wed, Jul 9, 2014 at 8:54 AM, Hotec04 <ha...@dish.com> wrote:
> Hi All,
>
> I tried to find apis for fuzzyRowFilter with range search but did not find
> any. Here is what I want to achieve: show the row key in a certain date
> range. Our row key design is like customer id + timestamp. Example is:
> 1234567892014-07-09T00:11:22.333-04:00. Input should be the start date and
> end date and output will be all row keys within that date period. Example:
> input start date = 2014-07-07 and end date = 2014-07-10 then output will be
> 1234567892014-07-09T00:11:22.333-04:00 and
> xxxxxxxxx2014-07-08Txx:xx:xx.xxx-04:00 which means all row keys regardless
> customer id within that date range should be returned.
>
> We are using hbase api 0.94.15-cdh4.6.0.jar and hbase is from cloudera
> manager. I am not sure whether they support hbase 0.99 api.
>
> Any help will be appreciated.
>
> Thank you!
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/How-to-perform-fuzzyRowFilter-with-range-search-tp4061159.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Re: How to perform fuzzyRowFilter with range search
Posted by Jonathan Hsieh <jo...@cloudera.com>.
FYI, CDH5 is based off of hbase 0.96.1.1.
On Wed, Jul 9, 2014 at 8:54 AM, Hotec04 <ha...@dish.com> wrote:
> Hi All,
>
> I tried to find apis for fuzzyRowFilter with range search but did not find
> any. Here is what I want to achieve: show the row key in a certain date
> range. Our row key design is like customer id + timestamp. Example is:
> 1234567892014-07-09T00:11:22.333-04:00. Input should be the start date and
> end date and output will be all row keys within that date period. Example:
> input start date = 2014-07-07 and end date = 2014-07-10 then output will be
> 1234567892014-07-09T00:11:22.333-04:00 and
> xxxxxxxxx2014-07-08Txx:xx:xx.xxx-04:00 which means all row keys regardless
> customer id within that date range should be returned.
>
> We are using hbase api 0.94.15-cdh4.6.0.jar and hbase is from cloudera
> manager. I am not sure whether they support hbase 0.99 api.
>
> Any help will be appreciated.
>
> Thank you!
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/How-to-perform-fuzzyRowFilter-with-range-search-tp4061159.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
--
// Jonathan Hsieh (shay)
// HBase Tech Lead, Software Engineer, Cloudera
// jon@cloudera.com // @jmhsieh
Re: How to perform fuzzyRowFilter with range search
Posted by Hotec04 <ha...@dish.com>.
Searched on cloudera, looks like they only support up to 0.94.16.
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/How-to-perform-fuzzyRowFilter-with-range-search-tp4061159p4061160.html
Sent from the HBase User mailing list archive at Nabble.com.