You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Andre Reiter <a....@web.de> on 2011/07/14 21:52:05 UTC

data structure

Hi everybody,

we have our hadoop + hbase cluster running at the moment with 6 servers

everything is working just fine. We have a web application, where data is stored with the row key = user id (meaningless UUID). So our users have a cookie, which is the row key, behind this key are families with items, i.e. family "impressions", where every impression is stored with its time stamp etc...

the row key is defined with the user id, to make the real time request possible, so we can retrieve all user data very fast

new we are running mapreduce jobs, to generate a report: for example we want to know how many impressions were done by all users in last x days. therefore the scan of the MR job is running over all data in our hbase table for the particular family. this takes at the moment about 70 seconds, which is actually a bit too long, and with the data growing, the time will increase, unless we add new workers to the cluster. we have right now 22 regions

the problem i see, is that we can not define a filter for the scan, the row key (user id) is just an UUID, nothing meaningfull in it

what can we do, to however improve (accelerate) the scan process? is it maybe advisable to store the data more redundant. so for example we create second table and store every impression twice, one time with the user id as row key in the first table, and the second one with a time stamp as a row key in the second table.
the data volume would grow twice as fast, but our scans will work x times faster on the second table compared to now

comments are very appreciated

andre


Re: data structure

Posted by Andre Reiter <a....@web.de>.
Stack wrote:
> On Thu, Jul 14, 2011 at 12:52 PM, Andre Reiter<a....@web.de>  wrote:
>
> Why is 70 seconds too long for a report?  70 seconds seems like a
> short mapreduce job (to me).
>
> You don't have that many regions.
>
> How fast would you like this operation to complete in?
>
> The report you describe above is predicated on looking at all data,
> right?  If so, I'm not sure how you'd avoid the job taking longer the
> more data you have (unless you up the parallelism and/or cluster size)
>

ok, 70 seconds is not really long, but, like i said, the data is growing, and with it the process time
we would like to have it running in, lets say 30 seconds, that would be nice :-)

>
> What do you want to filter out?  When you scan, you are working to
> narrow its scope by setting time-range, famliy, etc.
>
the filter by setting time-range, is exactly the thing, we would like set, so there would remain only a couple of interesting regions.
the time stamp is stored in the cell, but the filter can be defined on the key only, right?
  


Re: data structure

Posted by Stack <st...@duboce.net>.
On Thu, Jul 14, 2011 at 12:52 PM, Andre Reiter <a....@web.de> wrote:
> new we are running mapreduce jobs, to generate a report: for example we want
> to know how many impressions were done by all users in last x days.
> therefore the scan of the MR job is running over all data in our hbase table
> for the particular family. this takes at the moment about 70 seconds, which
> is actually a bit too long, and with the data growing, the time will
> increase, unless we add new workers to the cluster. we have right now 22
> regions
>

Why is 70 seconds too long for a report?  70 seconds seems like a
short mapreduce job (to me).

You don't have that many regions.

How fast would you like this operation to complete in?

The report you describe above is predicated on looking at all data,
right?  If so, I'm not sure how you'd avoid the job taking longer the
more data you have (unless you up the parallelism and/or cluster size)

> the problem i see, is that we can not define a filter for the scan, the row
> key (user id) is just an UUID, nothing meaningfull in it
>

What do you want to filter out?  When you scan, you are working to
narrow its scope by setting time-range, famliy, etc.


> what can we do, to however improve (accelerate) the scan process? is it
> maybe advisable to store the data more redundant. so for example we create
> second table and store every impression twice, one time with the user id as
> row key in the first table, and the second one with a time stamp as a row
> key in the second table.

You could do this to make a view that was more amenable to your report
generation.


> the data volume would grow twice as fast, but our scans will work x times
> faster on the second table compared to now
>

You'd have to figure what you can tolerate.  Slower writing because
now you are writing two places instead of one but your reports will
run faster.

St.Ack

Re: data structure

Posted by Claudio Martella <cl...@tis.bz.it>.
You can still trough a scanner defining the starting key in a smart way


On 7/15/11 3:39 PM, Andre Reiter wrote:
> hi Claudio,
> thanks for the hint
>
> the point is, that we need a fast request to the user data, that is
> why we need the row key to be the user_id
>
>
>
>> ----- Original Message -----
>> From: Claudio Martella <cl...@tis.bz.it>
>> Sent: Fri Jul 15 2011 14:40:38 GMT+0200 (CET)
>> To:
>> CC:
>> Subject: Re: data structure
>
>> supposed you want a per-hour granularity, you could have a key like this
>>
>> <userID>_<YYMMDDHH>
>>
>> where
>>
>> HH: hour of the day (0-23)
>> DD: day of the month
>> MM: month of the year
>> YY: year
>>
>>
>> this could be one way.
>>
>
>
>


-- 
Claudio Martella
Free Software & Open Technologies
Analyst

TIS innovation park
Via Siemens 19 | Siemensstr. 19
39100 Bolzano | 39100 Bozen
Tel. +39 0471 068 123
Fax  +39 0471 068 129
claudio.martella@tis.bz.it http://www.tis.bz.it

Short information regarding use of personal data. According to Section 13 of Italian Legislative Decree no. 196 of 30 June 2003, we inform you that we process your personal data in order to fulfil contractual and fiscal obligations and also to send you information regarding our services and events. Your personal data are processed with and without electronic means and by respecting data subjects' rights, fundamental freedoms and dignity, particularly with regard to confidentiality, personal identity and the right to personal data protection. At any time and without formalities you can write an e-mail to privacy@tis.bz.it in order to object the processing of your personal data for the purpose of sending advertising materials and also to exercise the right to access personal data and other rights referred to in Section 7 of Decree 196/2003. The data controller is TIS Techno Innovation Alto Adige, Siemens Street n. 19, Bolzano. You can find the complete information on the web site www.tis.bz.it.





Re: data structure

Posted by Andre Reiter <a....@web.de>.
hi Claudio,
thanks for the hint

the point is, that we need a fast request to the user data, that is why we need the row key to be the user_id



> ----- Original Message -----
> From: Claudio Martella <cl...@tis.bz.it>
> Sent: Fri Jul 15 2011 14:40:38 GMT+0200 (CET)
> To:
> CC:
> Subject: Re: data structure

> supposed you want a per-hour granularity, you could have a key like this
>
> <userID>_<YYMMDDHH>
>
> where
>
> HH: hour of the day (0-23)
> DD: day of the month
> MM: month of the year
> YY: year
>
>
> this could be one way.
>



Re: data structure

Posted by Claudio Martella <cl...@tis.bz.it>.
supposed you want a per-hour granularity, you could have a key like this

<userID>_<YYMMDDHH>

where

HH: hour of the day (0-23)
DD: day of the month
MM: month of the year
YY: year


this could be one way.

On 7/14/11 11:23 PM, Andre Reiter wrote:
>> ----- Original Message -----
>> From: Ted Dunning <td...@maprtech.com>
>> Sent: Thu Jul 14 2011 23:17:20 GMT+0200 (CET)
>> To:
>> CC:
>> Subject: Re: data structure
>
>> You can play tricks with the arrangement of the key.
>>
>> For instance, you can put date at the end of the key.  That would let
>> you
>> pull data for a particular user for a particular date range.  The date
>> should not be a time stamp, but should be a low-res version of time
>> (day-level resolution might be ok) so that you can minimize number of
>> rows.
>>
>
> hi Ted
>
> i do not see any way to put the date into the row key, there are many
> requests/impressions for a particular user (row key)
> so no chance to put a date into the key
>
> andre
>
>
>
>


-- 
Claudio Martella
Free Software & Open Technologies
Analyst

TIS innovation park
Via Siemens 19 | Siemensstr. 19
39100 Bolzano | 39100 Bozen
Tel. +39 0471 068 123
Fax  +39 0471 068 129
claudio.martella@tis.bz.it http://www.tis.bz.it

Short information regarding use of personal data. According to Section 13 of Italian Legislative Decree no. 196 of 30 June 2003, we inform you that we process your personal data in order to fulfil contractual and fiscal obligations and also to send you information regarding our services and events. Your personal data are processed with and without electronic means and by respecting data subjects' rights, fundamental freedoms and dignity, particularly with regard to confidentiality, personal identity and the right to personal data protection. At any time and without formalities you can write an e-mail to privacy@tis.bz.it in order to object the processing of your personal data for the purpose of sending advertising materials and also to exercise the right to access personal data and other rights referred to in Section 7 of Decree 196/2003. The data controller is TIS Techno Innovation Alto Adige, Siemens Street n. 19, Bolzano. You can find the complete information on the web site www.tis.bz.it.





Re: data structure

Posted by Ted Dunning <td...@maprtech.com>.
Put all of the requests for a particular date together.

Today, you update the value for a particular user by appending to the
current value.  All you need to do is to change that slightly by appending
new data to the value for the user + the current date.

If you want to migrate your database gently, you can even start by having
the update find old style keys, do the old-style update and also split the
appended data into several new style keys.  When you finish writing all the
new style keys with the segmented data, delete the original.  If the update
fails before writing all the new keys, the next update will fix things.
 Updates should fail only rarely so you would have a very small number of
users with both kinds of keys.

It would be better to do a cataclysmic conversion using a map-reduce
program, but that may not be feasible.

On Thu, Jul 14, 2011 at 2:23 PM, Andre Reiter <a....@web.de> wrote:

> ----- Original Message -----
>> From: Ted Dunning <td...@maprtech.com>
>> Sent: Thu Jul 14 2011 23:17:20 GMT+0200 (CET)
>> To:
>> CC:
>> Subject: Re: data structure
>>
>
>  You can play tricks with the arrangement of the key.
>>
>> For instance, you can put date at the end of the key.  That would let you
>> pull data for a particular user for a particular date range.  The date
>> should not be a time stamp, but should be a low-res version of time
>> (day-level resolution might be ok) so that you can minimize number of
>> rows.
>>
>>
> hi Ted
>
> i do not see any way to put the date into the row key, there are many
> requests/impressions for a particular user (row key)
> so no chance to put a date into the key
>
> andre
>
>
>
>

Re: data structure

Posted by Andre Reiter <a....@web.de>.
> ----- Original Message -----
> From: Ted Dunning <td...@maprtech.com>
> Sent: Thu Jul 14 2011 23:17:20 GMT+0200 (CET)
> To:
> CC:
> Subject: Re: data structure

> You can play tricks with the arrangement of the key.
>
> For instance, you can put date at the end of the key.  That would let you
> pull data for a particular user for a particular date range.  The date
> should not be a time stamp, but should be a low-res version of time
> (day-level resolution might be ok) so that you can minimize number of rows.
>

hi Ted

i do not see any way to put the date into the row key, there are many requests/impressions for a particular user (row key)
so no chance to put a date into the key

andre




Re: data structure

Posted by Ted Dunning <td...@maprtech.com>.
You can play tricks with the arrangement of the key.

For instance, you can put date at the end of the key.  That would let you
pull data for a particular user for a particular date range.  The date
should not be a time stamp, but should be a low-res version of time
(day-level resolution might be ok) so that you can minimize number of rows.

On Thu, Jul 14, 2011 at 12:52 PM, Andre Reiter <a....@web.de> wrote:

> Hi everybody,
>
> we have our hadoop + hbase cluster running at the moment with 6 servers
>
> everything is working just fine. We have a web application, where data is
> stored with the row key = user id (meaningless UUID). So our users have a
> cookie, which is the row key, behind this key are families with items, i.e.
> family "impressions", where every impression is stored with its time stamp
> etc...
>
> the row key is defined with the user id, to make the real time request
> possible, so we can retrieve all user data very fast
>
> new we are running mapreduce jobs, to generate a report: for example we
> want to know how many impressions were done by all users in last x days.
> therefore the scan of the MR job is running over all data in our hbase table
> for the particular family. this takes at the moment about 70 seconds, which
> is actually a bit too long, and with the data growing, the time will
> increase, unless we add new workers to the cluster. we have right now 22
> regions
>
> the problem i see, is that we can not define a filter for the scan, the row
> key (user id) is just an UUID, nothing meaningfull in it
>
> what can we do, to however improve (accelerate) the scan process? is it
> maybe advisable to store the data more redundant. so for example we create
> second table and store every impression twice, one time with the user id as
> row key in the first table, and the second one with a time stamp as a row
> key in the second table.
> the data volume would grow twice as fast, but our scans will work x times
> faster on the second table compared to now
>
> comments are very appreciated
>
> andre
>
>

Re: data structure

Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi Andre,
 
In the process of developing some of our HBase-based products we've built a generic aggregation framework that is nice, flexible and extensible, and it sounds like it could build those reports you are after.  We run MR aggregate job that reads raw data (e.g. your impression data) from either HDFS or HBase (we write our data to HBase via Flume), computes aggregates of various kinds (defined in a config), and stores those aggregates back in HBase, from which a web front-end can retrieve them very quickly, while filtering them by various criteria.  I'm not exactly sure what takes 70 seconds in your case - aggregation of raw data or retrieving data to generate reports?  If it's the former, 70 seconds sounds OK and acceptable, but not if it's the latter, in which case see above how we do it.  I hope this helps a bit.

Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Hadoop - HBase
Hadoop ecosystem search :: http://search-hadoop.com/


>________________________________
>From: Andre Reiter <a....@web.de>
>To: user@hbase.apache.org
>Sent: Thursday, July 14, 2011 3:52 PM
>Subject: data structure
>
>Hi everybody,
>
>we have our hadoop + hbase cluster running at the moment with 6 servers
>
>everything is working just fine. We have a web application, where data is stored with the row key = user id (meaningless UUID). So our users have a cookie, which is the row key, behind this key are families with items, i.e. family "impressions", where every impression is stored with its time stamp etc...
>
>the row key is defined with the user id, to make the real time request possible, so we can retrieve all user data very fast
>
>new we are running mapreduce jobs, to generate a report: for example we want to know how many impressions were done by all users in last x days. therefore the scan of the MR job is running over all data in our hbase table for the particular family. this takes at the moment about 70 seconds, which is actually a bit too long, and with the data growing, the time will increase, unless we add new workers to the cluster. we have right now 22 regions
>
>the problem i see, is that we can not define a filter for the scan, the row key (user id) is just an UUID, nothing meaningfull in it
>
>what can we do, to however improve (accelerate) the scan process? is it maybe advisable to store the data more redundant. so for example we create second table and store every impression twice, one time with the user id as row key in the first table, and the second one with a time stamp as a row key in the second table.
>the data volume would grow twice as fast, but our scans will work x times faster on the second table compared to now
>
>comments are very appreciated
>
>andre
>
>
>
>

Re: data structure

Posted by Ted Dunning <td...@maprtech.com>.
Averages are easy to rollup as well.

Rank statistics like median, min, max and quartiles are not much harder.

Total uniques are more difficult.  If you have decent distributional
information, these can be estimated reasonably well.

Mahout has code for the first two.

On Sun, Jul 17, 2011 at 9:30 AM, Arvind Jayaprakash <wo...@anomalizer.net>wrote:

> On Jul 14, Andre Reiter wrote:
> >new we are running mapreduce jobs, to generate a report: for example we
> >want to know how many impressions were done by all users in last x
> >days. therefore the scan of the MR job is running over all data in our
> >hbase table for the particular family. this takes at the moment about
> >70 seconds, which is actually a bit too long, and with the data
> >growing, the time will increase, unless we add new workers to the
> >cluster. we have right now 22 regions
>
> Are you looking for average number of impressions per user in the last
> 'x' days or total number of impressions across all users in the last 'x'
> days? I assume it is the latter.
>
> The only reasonable way is to do frequent rollups (think count for every
> minute/hour) and store it for future use. The cost of performing these
> rollups wil always be a function of your traffic/data. However, the cost
> of retrieving your answer should be fixed for a given 'x' and the size
> of the rollup window regardless of how much traffic you see. This way,
> your online application (I'm guessing from your latency needs) is
> de-linked from raw data volumes.
>

Re: data structure

Posted by Arvind Jayaprakash <wo...@anomalizer.net>.
On Jul 14, Andre Reiter wrote:
>new we are running mapreduce jobs, to generate a report: for example we
>want to know how many impressions were done by all users in last x
>days. therefore the scan of the MR job is running over all data in our
>hbase table for the particular family. this takes at the moment about
>70 seconds, which is actually a bit too long, and with the data
>growing, the time will increase, unless we add new workers to the
>cluster. we have right now 22 regions

Are you looking for average number of impressions per user in the last
'x' days or total number of impressions across all users in the last 'x'
days? I assume it is the latter.

The only reasonable way is to do frequent rollups (think count for every
minute/hour) and store it for future use. The cost of performing these
rollups wil always be a function of your traffic/data. However, the cost
of retrieving your answer should be fixed for a given 'x' and the size
of the rollup window regardless of how much traffic you see. This way,
your online application (I'm guessing from your latency needs) is
de-linked from raw data volumes.

Re: data structure

Posted by Andre Reiter <a....@web.de>.
> ----- Original Message -----
> From: Doug Meil <do...@explorysmedical.com>
> Sent: Thu Jul 14 2011 22:29:16 GMT+0200 (CET)
> To:
> CC:
> Subject: Re: data structure

>
> Hi there-
>
> A few high-level suggestions...
>
> re:  "to generate a report: for example we want to know how many
> impressions were done by all users in last x days"
>
> Can you create a summary table by day (via MR job), and then have your
> ad-hoc report hit the summary table?
>
> Re:  "and with the data growing, the time will increase"
>
>
> Yes.  As you add more and more data processing times will slow.  That's
> why you need to expect to periodically expand your cluster.
>
>

i guess a summary table will be it
the only disadvantage of such tables is, that its not that flexible
so ie if i store data for every hour (24 entries a day), i can run fast reports for special  time ranges, ie 12:00 to 15:00

but there is no way to generate a report for the time range 12:30 to 13:45

i guess we will live with that constraint

i thought, the hadoop+hbase+mapreduce is such a cool magic stuff, there is not need for summary tables, and data scans are running withins milliseconds... ;-)



Re: data structure

Posted by Doug Meil <do...@explorysmedical.com>.
Hi there-

A few high-level suggestions...

re:  "to generate a report: for example we want to know how many
impressions were done by all users in last x days"

Can you create a summary table by day (via MR job), and then have your
ad-hoc report hit the summary table?

Re:  "and with the data growing, the time will increase"


Yes.  As you add more and more data processing times will slow.  That's
why you need to expect to periodically expand your cluster.



On 7/14/11 3:52 PM, "Andre Reiter" <a....@web.de> wrote:

>Hi everybody,
>
>we have our hadoop + hbase cluster running at the moment with 6 servers
>
>everything is working just fine. We have a web application, where data is
>stored with the row key = user id (meaningless UUID). So our users have a
>cookie, which is the row key, behind this key are families with items,
>i.e. family "impressions", where every impression is stored with its time
>stamp etc...
>
>the row key is defined with the user id, to make the real time request
>possible, so we can retrieve all user data very fast
>
>new we are running mapreduce jobs, to generate a report: for example we
>want to know how many impressions were done by all users in last x days.
>therefore the scan of the MR job is running over all data in our hbase
>table for the particular family. this takes at the moment about 70
>seconds, which is actually a bit too long, and with the data growing, the
>time will increase, unless we add new workers to the cluster. we have
>right now 22 regions
>
>the problem i see, is that we can not define a filter for the scan, the
>row key (user id) is just an UUID, nothing meaningfull in it
>
>what can we do, to however improve (accelerate) the scan process? is it
>maybe advisable to store the data more redundant. so for example we
>create second table and store every impression twice, one time with the
>user id as row key in the first table, and the second one with a time
>stamp as a row key in the second table.
>the data volume would grow twice as fast, but our scans will work x times
>faster on the second table compared to now
>
>comments are very appreciated
>
>andre
>