You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Andre Reiter <a....@web.de> on 2010/10/13 14:57:25 UTC
Architecture
Hello everybody,
i'm evaluating hadoop for a new platform.
the application is a web based application, and the challange is to handle
requests up to 1.000.000 times per second. this can actually be solved by
load ballanced webservers.
The problem now is to persist user data at real time. Let's say every
request appends some data to the user record, i.e. 100 bytes are appended.
The record would have an identifier placed in the cookie.
The problem now is to look up the user record for every request at runtime.
Is it possible to get the user record with the information about all former
requests in a short time i.e. 50 ms ???
i hove no experience with hadoop yet
every help would be very appreciated
thanks in advance
best regards
andre
Re: Architecture
Posted by Owen O'Malley <om...@apache.org>.
Here is a presentation from Hadoop Summit 2009 "HBase goes Realtime"
that gives numbers for latency with HBase. Redirecting to common-user.
http://bit.ly/aJEwYj
-- Owen
Re: Architecture
Posted by Owen O'Malley <om...@apache.org>.
Here is a presentation from Hadoop Summit 2009 "HBase goes Realtime"
that gives numbers for latency with HBase. Redirecting to common-user.
http://bit.ly/aJEwYj
-- Owen
Re: Architecture
Posted by Lars Francke <la...@gmail.com>.
> Not to sound argumentative but because perhaps I have misunderstood, Hbase doesn't seem well suited to realtime requests at all. Perhaps you could elaborate on the architecture you would use to support this application?
Quite the opposite. HBase is all about realtime requests. It's
optimized for it. Here's the description from the homepage:
"HBase is the Hadoop database. Use it when you need random, realtime
read/write access to your Big Data. This project's goal is the hosting
of very large tables -- billions of rows X millions of columns -- atop
clusters of commodity hardware.
Cheers,
Lars
Re: Architecture
Posted by daniel sikar <ds...@gmail.com>.
Perhaps in-memory data grids could be considered?
On 13 October 2010 17:02, Jonathan Creasy <jo...@announcemedia.com> wrote:
> Not to sound argumentative but because perhaps I have misunderstood, Hbase doesn't seem well suited to realtime requests at all. Perhaps you could elaborate on the architecture you would use to support this application?
>
> data starts here, goes here, this is done, it is accessed by a webserver here, etc
>
> Thank you!
>
> On Oct 13, 2010, at 8:10 AM, "Tim Robertson" <ti...@gmail.com> wrote:
>
>> This sounds like you should be looking to HBase. HBase is a data
>> store that builds on top of Hadoop to meet those kind of real time
>> storage and retrieval requirements.
>>
>> HTH,
>> Tim
>>
>> On Wed, Oct 13, 2010 at 2:57 PM, Andre Reiter <a....@web.de> wrote:
>>> Hello everybody,
>>>
>>> i'm evaluating hadoop for a new platform.
>>> the application is a web based application, and the challange is to handle
>>> requests up to 1.000.000 times per second. this can actually be solved by
>>> load ballanced webservers.
>>> The problem now is to persist user data at real time. Let's say every
>>> request appends some data to the user record, i.e. 100 bytes are appended.
>>> The record would have an identifier placed in the cookie.
>>>
>>> The problem now is to look up the user record for every request at runtime.
>>> Is it possible to get the user record with the information about all former
>>> requests in a short time i.e. 50 ms ???
>>>
>>> i hove no experience with hadoop yet
>>>
>>> every help would be very appreciated
>>> thanks in advance
>>>
>>> best regards
>>> andre
>>>
>>>
>>>
>
Re: Architecture
Posted by Jonathan Creasy <jo...@announcemedia.com>.
Not to sound argumentative but because perhaps I have misunderstood, Hbase doesn't seem well suited to realtime requests at all. Perhaps you could elaborate on the architecture you would use to support this application?
data starts here, goes here, this is done, it is accessed by a webserver here, etc
Thank you!
On Oct 13, 2010, at 8:10 AM, "Tim Robertson" <ti...@gmail.com> wrote:
> This sounds like you should be looking to HBase. HBase is a data
> store that builds on top of Hadoop to meet those kind of real time
> storage and retrieval requirements.
>
> HTH,
> Tim
>
> On Wed, Oct 13, 2010 at 2:57 PM, Andre Reiter <a....@web.de> wrote:
>> Hello everybody,
>>
>> i'm evaluating hadoop for a new platform.
>> the application is a web based application, and the challange is to handle
>> requests up to 1.000.000 times per second. this can actually be solved by
>> load ballanced webservers.
>> The problem now is to persist user data at real time. Let's say every
>> request appends some data to the user record, i.e. 100 bytes are appended.
>> The record would have an identifier placed in the cookie.
>>
>> The problem now is to look up the user record for every request at runtime.
>> Is it possible to get the user record with the information about all former
>> requests in a short time i.e. 50 ms ???
>>
>> i hove no experience with hadoop yet
>>
>> every help would be very appreciated
>> thanks in advance
>>
>> best regards
>> andre
>>
>>
>>
Re: Architecture
Posted by Tim Robertson <ti...@gmail.com>.
This sounds like you should be looking to HBase. HBase is a data
store that builds on top of Hadoop to meet those kind of real time
storage and retrieval requirements.
HTH,
Tim
On Wed, Oct 13, 2010 at 2:57 PM, Andre Reiter <a....@web.de> wrote:
> Hello everybody,
>
> i'm evaluating hadoop for a new platform.
> the application is a web based application, and the challange is to handle
> requests up to 1.000.000 times per second. this can actually be solved by
> load ballanced webservers.
> The problem now is to persist user data at real time. Let's say every
> request appends some data to the user record, i.e. 100 bytes are appended.
> The record would have an identifier placed in the cookie.
>
> The problem now is to look up the user record for every request at runtime.
> Is it possible to get the user record with the information about all former
> requests in a short time i.e. 50 ms ???
>
> i hove no experience with hadoop yet
>
> every help would be very appreciated
> thanks in advance
>
> best regards
> andre
>
>
>
RE: Architecture
Posted by Anil Uberoi <an...@kitenga.com>.
Andre,
Hope all is well.
I 'd like to invite you to check out www.kitenga.com
We'd be more than happy to help.
Best regards,
Anil
------------------------------------------------------------------
Anil Uberoi | CEO | Kitenga | Reinventing Information
www.kitenga.com | 510.507.3399
-----Original Message-----
From: a.reiter@web.de [mailto:a.reiter@web.de]
Sent: Wednesday, October 13, 2010 5:57 AM
To: general@hadoop.apache.org
Subject: Architecture
Hello everybody,
i'm evaluating hadoop for a new platform.
the application is a web based application, and the challange is to handle
requests up to 1.000.000 times per second. this can actually be solved by
load ballanced webservers.
The problem now is to persist user data at real time. Let's say every
request appends some data to the user record, i.e. 100 bytes are appended.
The record would have an identifier placed in the cookie.
The problem now is to look up the user record for every request at runtime.
Is it possible to get the user record with the information about all former
requests in a short time i.e. 50 ms ???
i hove no experience with hadoop yet
every help would be very appreciated
thanks in advance
best regards
andre