You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Michael Hauck <mi...@maxviva.com> on 2009/07/09 11:41:37 UTC
MapReduce newbie questions
Hi,
i'm new to hbase MapReduce and want to do following:
- create daily statistics with sql queries against a sql database
- store statistic results in hbase
- run daily MapReduce on that results to compute monthly statistics
I stored this data in hbase table 'route_conversion_statistics'.
My keys have the format '<agency_id>_yyyy-MM-dd' like '208_2009-06-08'
My ColumnFamilies are 'looks', 'bookings', 'turnover', 'paxcount'
For example:
The row '208_2009-06-08' has about 30000 column like this:
looks:FRAMUC, value=123
looks:FRALAX, value=456
...
bookings:FRAMUC, value=15
bookings:FRALAX, value=34
...
turnover:FRAMUC, value=1534.34
turnover:FRALAX, value=4574.35
...
paxcount:FRAMUC, value=356
paxcount:FRALAX, value=5676
...
Now i want to create a new row with the corresponding key '208_2009-06'
and put
the sum of all columns from '208_2009-06-01' to '208_2009-06-30'
What is the best practice to do this?
How can i scan over this monthly range?
I use hadoop 0.19.1 and HBase 0.19.1
Thanks,
Michael
--
Michael Hauck <mi...@maxviva.com>
Re: MapReduce newbie questions
Posted by stack <st...@duboce.net>.
On Fri, Jul 10, 2009 at 2:10 AM, Michael Hauck <mi...@maxviva.com>wrote:
> Thanks for your help.
>
> > First, I recommend upgrading to the latest HBase 0.19 release, 0.19.3.
> >
> Do i have to migrate/convert/import the data ?
>
No.
Just ensure you replace all of your old instances because mistakenly changed
the IPC version number between 0.19.0 and 0.19.3. If you don't upgrade all,
you'll run into connection issues because old can't communicate with the
new.
>
> ....
>
> > There's an example in TIFB docs:
> >
> >
> http://hadoop.apache.org/hbase/docs/r0.19.3/api/org/apache/hadoop/hbase/mapred/TableInputFormatBase.html
> >
> The are the inputColumns in this example only the family names or do
> they have to be fully qualified?
>
Not sure I understand the question. Fully-qualified will work and you
probably want to use these rather than family only. Regexes might work for
column names too. Check the code.
St.Ack
Re: MapReduce newbie questions
Posted by Michael Hauck <mi...@maxviva.com>.
Thanks for your help.
> First, I recommend upgrading to the latest HBase 0.19 release, 0.19.3.
>
Do i have to migrate/convert/import the data ?
> You have a few choices, but in short you want to use filters.
>
> http://hadoop.apache.org/hbase/docs/r0.19.3/api/org/apache/hadoop/hbase/filter/package-summary.html
>
> Specifically, you should look at the RegExpRowFilter:
>
> http://hadoop.apache.org/hbase/docs/r0.19.3/api/org/apache/hadoop/hbase/filter/RegExpRowFilter.html
>
Should be handy.
>
> You could set up the regular expression to only return stuff from the
> month you want. Inside the MR job you would know every row returned
> would come from the month in question and would be able to look at the
> key to determine the agency_id and day.
>
> There's an example in TIFB docs:
>
> http://hadoop.apache.org/hbase/docs/r0.19.3/api/org/apache/hadoop/hbase/mapred/TableInputFormatBase.html
>
The are the inputColumns in this example only the family names or do
they have to be fully qualified?
--
Michael Hauck <mi...@maxviva.com>
Re: MapReduce newbie questions
Posted by Jonathan Gray <jl...@streamy.com>.
First, I recommend upgrading to the latest HBase 0.19 release, 0.19.3.
You have a few choices, but in short you want to use filters.
http://hadoop.apache.org/hbase/docs/r0.19.3/api/org/apache/hadoop/hbase/filter/package-summary.html
Specifically, you should look at the RegExpRowFilter:
http://hadoop.apache.org/hbase/docs/r0.19.3/api/org/apache/hadoop/hbase/filter/RegExpRowFilter.html
You could set up the regular expression to only return stuff from the
month you want. Inside the MR job you would know every row returned
would come from the month in question and would be able to look at the
key to determine the agency_id and day.
There's an example in TIFB docs:
http://hadoop.apache.org/hbase/docs/r0.19.3/api/org/apache/hadoop/hbase/mapred/TableInputFormatBase.html
Best of luck!
JG
Michael Hauck wrote:
> Hi,
>
> i'm new to hbase MapReduce and want to do following:
>
> - create daily statistics with sql queries against a sql database
> - store statistic results in hbase
> - run daily MapReduce on that results to compute monthly statistics
>
> I stored this data in hbase table 'route_conversion_statistics'.
> My keys have the format '<agency_id>_yyyy-MM-dd' like '208_2009-06-08'
> My ColumnFamilies are 'looks', 'bookings', 'turnover', 'paxcount'
>
> For example:
> The row '208_2009-06-08' has about 30000 column like this:
>
> looks:FRAMUC, value=123
> looks:FRALAX, value=456
> ...
> bookings:FRAMUC, value=15
> bookings:FRALAX, value=34
> ...
> turnover:FRAMUC, value=1534.34
> turnover:FRALAX, value=4574.35
> ...
> paxcount:FRAMUC, value=356
> paxcount:FRALAX, value=5676
> ...
>
> Now i want to create a new row with the corresponding key '208_2009-06'
> and put
> the sum of all columns from '208_2009-06-01' to '208_2009-06-30'
>
> What is the best practice to do this?
> How can i scan over this monthly range?
>
>
> I use hadoop 0.19.1 and HBase 0.19.1
>
> Thanks,
> Michael