You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by aakash_j j_shah <aa...@yahoo.com> on 2008/12/20 02:05:42 UTC
Architecture question.
Hello All,
I am designing an architecture which should support 10 million records storage capacity and 1 million updates / minute. Data persistancy is not that important as I will be purging this data every day.
I am familiar with memcache but not hadoop. It will be great if I can
get some points from the group regarding designing this architecture.
Thanks,
Aakash.
Re: Architecture question.
Posted by tim robertson <ti...@gmail.com>.
I would also consider a DB for this... 10M and 2 columns is not a lot
of data so I would look to have it in memory with some DB index or
memory hash for querying.
(We are keeping the indexes of tables with 150M records, 30M and 10M
and joining between them with around 25 indexes on the 150M table - on
a single machine with 64G ram - mysql)
I would spend a day trying some testing on H2 and mysql in myisam mode
which is a fast DB or try their in-memory DB - we have seen 600
"query+update" operations per second at peak - so this is only 3
machines to do 10M in 10 minutes. H2 in embedded mode is also
impressively fast and I think there is a memory only mode also.
Are you able to build the new version offline?
Have you considered writing the new version to simple CSV file and
loading into a DB. This is a very fast operation also.
Make sure you use the smallest possible type you can for the data to
improve performance - if a TINYINT is enough, then use it. The
difference will be huge.
It also depends on how you query your data as to what indexes you need
built and what the load will be - perhaps you can describe your
problem more?
Cheers
Tim
On Wed, Dec 24, 2008 at 11:45 AM, Steve Loughran <st...@apache.org> wrote:
> aakash shah wrote:
>>
>> We can assume that this record has only one key->value mapping. Value will
>> be updated every minute. Currently we have 1 Million these ( key->value )
>> pairs but I have to make sure that we can scale it upto 10 million of these
>> ( key-> value ) pairs.
>>
>> Every 10 minute I will be updating all of these value using their keys.
>> This is the reason I cannot go for database as a solution.
>
> I wouldn't be so quick to dismiss a database. All your big telcos run their
> mobile phone systems on databases, where the big issue is having enough
> memory for the DB to stay in memory; some dedicated databases (e.g.
> TimesTen) are designed to have bounded latency on lookup so you can predict
> how long operations will take.
>
> That said, if you are only doing atomic updates of a single record, there's
> less need for the advanced features. Assuming >1 machine, some kind of
> distributed hash table may work
>
>
>> I was thinking about going with memcache pool. In the mean-time I heard
>> about hadoop and wanted to get advice from this mailing list regarding
>> memcache pool vs hadoop for this specific problem.
>
> It's not an area Hadoop deals with at all.
>
Re: Architecture question.
Posted by Bill de hOra <bi...@dehora.net>.
Steve Loughran wrote:
> aakash shah wrote:
>> We can assume that this record has only one key->value mapping. Value
>> will be updated every minute. Currently we have 1 Million these (
>> key->value ) pairs but I have to make sure that we can scale it upto
>> 10 million of these ( key-> value ) pairs.
>>
>> Every 10 minute I will be updating all of these value using their
>> keys. This is the reason I cannot go for database as a solution.
>
> I wouldn't be so quick to dismiss a database. All your big telcos run
> their mobile phone systems on databases, where the big issue is having
> enough memory for the DB to stay in memory; some dedicated databases
> (e.g. TimesTen) are designed to have bounded latency on lookup so you
> can predict how long operations will take.
>
> That said, if you are only doing atomic updates of a single record,
> there's less need for the advanced features. Assuming >1 machine, some
> kind of distributed hash table may work
>
>
>> I was thinking about going with memcache pool. In the mean-time I
>> heard about hadoop and wanted to get advice from this mailing list
>> regarding memcache pool vs hadoop for this specific problem.
>
> It's not an area Hadoop deals with at all.
The record size sounds too small for HDFS, unless the records are in
turn grouped to something optimal for the block size. For records that
size, I would also consider a) writing them out again instead of doing
updates, b) testing for physical (disk) bottlenecks.
Also, there's memcacheddb as an alternative for persistent hashing:
http://memcachedb.org/benchmark.html
Bill
Re: Architecture question.
Posted by Steve Loughran <st...@apache.org>.
aakash shah wrote:
> We can assume that this record has only one key->value mapping. Value will be updated every minute. Currently we have 1 Million these ( key->value ) pairs but I have to make sure that we can scale it upto 10 million of these ( key-> value ) pairs.
>
> Every 10 minute I will be updating all of these value using their keys. This is the reason I cannot go for database as a solution.
I wouldn't be so quick to dismiss a database. All your big telcos run
their mobile phone systems on databases, where the big issue is having
enough memory for the DB to stay in memory; some dedicated databases
(e.g. TimesTen) are designed to have bounded latency on lookup so you
can predict how long operations will take.
That said, if you are only doing atomic updates of a single record,
there's less need for the advanced features. Assuming >1 machine, some
kind of distributed hash table may work
> I was thinking about going with memcache pool. In the mean-time I heard about hadoop and wanted to get advice from this mailing list regarding memcache pool vs hadoop for this specific problem.
It's not an area Hadoop deals with at all.
Re: Architecture question.
Posted by aakash shah <aa...@yahoo.com>.
We can assume that this record has only one key->value mapping. Value will be updated every minute. Currently we have 1 Million these ( key->value ) pairs but I have to make sure that we can scale it upto 10 million of these ( key-> value ) pairs.
Every 10 minute I will be updating all of these value using their keys. This is the reason I cannot go for database as a solution.
I was thinking about going with memcache pool. In the mean-time I heard about hadoop and wanted to get advice from this mailing list regarding memcache pool vs hadoop for this specific problem.
Thanks,
Aakash.
--- On Mon, 12/22/08, Steve Loughran <st...@apache.org> wrote:
From: Steve Loughran <st...@apache.org>
Subject: Re: Architecture question.
To: core-user@hadoop.apache.org
Date: Monday, December 22, 2008, 5:36 AM
aakash_j j_shah wrote:
> Hello Edwin,
> Thanks for the answer. Records are very small usually key is about 64
bytes ( ascii ) and updates are for 10 integer values. So I would say that
record size including key is about 104 bytes.
Phone numbers? and device IDs? And you want do number->ID or ID->number
mappings?
this is more of a database problem than a filesystem one
Re: Architecture question.
Posted by Steve Loughran <st...@apache.org>.
aakash_j j_shah wrote:
> Hello Edwin,
>
> Thanks for the answer. Records are very small usually key is about 64 bytes ( ascii ) and updates are for 10 integer values. So I would say that record size including key is about 104 bytes.
>
Phone numbers? and device IDs? And you want do number->ID or ID->number
mappings?
this is more of a database problem than a filesystem one
Re: Architecture question.
Posted by aakash_j j_shah <aa...@yahoo.com>.
Hello Edwin,
Thanks for the answer. Records are very small usually key is about 64 bytes ( ascii ) and updates are for 10 integer values. So I would say that record size including key is about 104 bytes.
Sid.
--- On Fri, 12/19/08, Edwin Gonzalez <go...@zenbe.com> wrote:
From: Edwin Gonzalez <go...@zenbe.com>
Subject: Re: Architecture question.
To: core-user@hadoop.apache.org, aakash_j_shah@yahoo.com
Date: Friday, December 19, 2008, 5:13 PM
How large are the records ?
1 mil updates / mil. . . . you mind sharing the complexity of the updates ?
On Fri, Dec 19, 2008 at 8:05 PM "aakash_j j_shah"
<aa...@yahoo.com> wrote:
>Hello All,
>
> I am designing an architecture which should support 10 million
records storage capacity and 1 million updates / minute. Data persistancy is not
that important as I will be purging this data every day.
>
>
> I am familiar with memcache but not hadoop. It will be great if I can
> get some points from the group regarding designing this architecture.
>
> Thanks,
> Aakash.
>
>
>
Re: Architecture question.
Posted by Edwin Gonzalez <go...@zenbe.com>.
How large are the records ?
1 mil updates / mil. . . . you mind sharing the complexity of the updates ?
On Fri, Dec 19, 2008 at 8:05 PM "aakash_j j_shah" <aa...@yahoo.com> wrote:
>Hello All,
>
> I am designing an architecture which should support 10 million records storage capacity and 1 million updates / minute. Data persistancy is not that important as I will be purging this data every day.
>
>
> I am familiar with memcache but not hadoop. It will be great if I can
> get some points from the group regarding designing this architecture.
>
> Thanks,
> Aakash.
>
>
>