You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Linton N <ga...@gmail.com> on 2010/04/16 08:28:33 UTC
Regarding Cassandra Scalability
hi ,
I am working for the past 1 year with hadoop, but quite new to
cassandra, I would like to get clarified few things regarding the
scalability of Cassandra. Can it scall up to TB of data ?
Please provide me some links regarding this..
--
--
With Love
Lin N
Re: Regarding Cassandra Scalability
Posted by Mark Robson <ma...@gmail.com>.
>
> On Fri, Apr 16, 2010 at 1:28 PM, Linton N <ga...@gmail.com>wrote:
>
>> hi ,
>> I am working for the past 1 year with hadoop, but quite new to
>> cassandra, I would like to get clarified few things regarding the
>> scalability of Cassandra. Can it scall up to TB of data ?
>>
>
TB of data is not really that much; laptops have hard drives this big
nowadays.
What is interesting is scaling beyond ~ 100Tb, which is the kind of size
that more conventional storage systems (e.g. single large node) get tricky.
Mark
Re: Regarding Cassandra Scalability
Posted by Peter Chang <pe...@gmail.com>.
The redundancy/denormalization takes advantage of cheap writes to make
reads really quick. Imagine a query that returns one row with your
whole tweet stream vs having to do 50 separate lookups per tweet.
Space is cheap and the upside is performance.... Especially if you're
getting a lot of fail whales. :)
Sent from my iPhone
On Apr 16, 2010, at 2:46 PM, Mike Gallamore <mike.e.gallamore@googlemail.com
> wrote:
> Also people with 1M followers tend to have "public" tweets, which
> means really I think it would be the same as subscribing to a RSS
> feed or whatever. You aren't getting a local copy because you will
> "always" have access to the tweet as will everyone else. Also tweets
> don't change AFAIK so no point in having redundant copies.
> On 04/16/2010 01:42 PM, Peter Chang wrote:
>> Yeah. I wasn't sure if Cassandra was optimized for binary data
>> especially since any site of that size will use a CDN. Interesting
>> read though.
>>
>> I think 1K per tweet is off by an order of magnitude considering they
>> only allow 140 characters. Regardless the number of users with> 1MM
>> is probably a handful. Also im guessing they purge data after a
>> certain window (like 30 days for example).
>>
>> Sent from my iPhone
>>
>>
>> On Apr 16, 2010, at 12:02 PM, gabriele renzi<rf...@gmail.com>
>> wrote:
>>
>>
>>> On Fri, Apr 16, 2010 at 6:41 PM, Peter Chang<pe...@gmail.com>
>>> wrote:
>>>
>>>> FB also does pics and movies so 1MB is way off depending on where
>>>> they
>>>> manage such binary data.
>>>>
>>> apparently not in cassandra http://www.facebook.com/note.php?note_id=76191543919
>>>
>>>
>>>> I do agree that 1MB of text alone is a lot of text
>>>> which is more relevant in the case of Twitter. The only large thing
>>>> you
>>>> leave out is denormalization. Every tweet you write is likely
>>>> denormalized
>>>> across your followers to allow for quick read access.
>>>>
>>> .. but considering many users have _millions_ of followers, this may
>>> be quite a bit more data. Assuming 1k per tweet, this would mean one
>>> from @aplusk (4.7M followers) would take more than 4 gigabytes of
>>> data. Assuming ten tweets a day, in one month he'd produce one TB.
>>>
>>> I'd say they only store references (increasing number lists can also
>>> be encoded very cleverly), or in some other way I'm not smart enough
>>> to think of.
>>>
>
Re: Regarding Cassandra Scalability
Posted by Ryan King <ry...@twitter.com>.
That's just the data our analytics team produces (logs, etc).
Production/online data is separate.
-ryan
On Fri, Apr 16, 2010 at 2:22 PM, Stu Hood <st...@rackspace.com> wrote:
> http://twitter.com/jromeh/status/12295736793
>
> -----Original Message-----
> From: "Mike Gallamore" <mi...@googlemail.com>
> Sent: Friday, April 16, 2010 3:46pm
> To: user@cassandra.apache.org
> Subject: Re: Regarding Cassandra Scalability
>
> Also people with 1M followers tend to have "public" tweets, which means
> really I think it would be the same as subscribing to a RSS feed or
> whatever. You aren't getting a local copy because you will "always" have
> access to the tweet as will everyone else. Also tweets don't change
> AFAIK so no point in having redundant copies.
> On 04/16/2010 01:42 PM, Peter Chang wrote:
>> Yeah. I wasn't sure if Cassandra was optimized for binary data
>> especially since any site of that size will use a CDN. Interesting
>> read though.
>>
>> I think 1K per tweet is off by an order of magnitude considering they
>> only allow 140 characters. Regardless the number of users with> 1MM
>> is probably a handful. Also im guessing they purge data after a
>> certain window (like 30 days for example).
>>
>> Sent from my iPhone
>>
>>
>> On Apr 16, 2010, at 12:02 PM, gabriele renzi<rf...@gmail.com> wrote:
>>
>>
>>> On Fri, Apr 16, 2010 at 6:41 PM, Peter Chang<pe...@gmail.com>
>>> wrote:
>>>
>>>> FB also does pics and movies so 1MB is way off depending on where
>>>> they
>>>> manage such binary data.
>>>>
>>> apparently not in cassandra http://www.facebook.com/note.php?note_id=76191543919
>>>
>>>
>>>> I do agree that 1MB of text alone is a lot of text
>>>> which is more relevant in the case of Twitter. The only large thing
>>>> you
>>>> leave out is denormalization. Every tweet you write is likely
>>>> denormalized
>>>> across your followers to allow for quick read access.
>>>>
>>> .. but considering many users have _millions_ of followers, this may
>>> be quite a bit more data. Assuming 1k per tweet, this would mean one
>>> from @aplusk (4.7M followers) would take more than 4 gigabytes of
>>> data. Assuming ten tweets a day, in one month he'd produce one TB.
>>>
>>> I'd say they only store references (increasing number lists can also
>>> be encoded very cleverly), or in some other way I'm not smart enough
>>> to think of.
>>>
>
>
>
>
Re: Regarding Cassandra Scalability
Posted by Mike Gallamore <mi...@googlemail.com>.
Does that include HD copies of CNN et al reading tweets to people on
T.V.? You know your medium is doomed when you're reduced to reading
comments from random_dude64 and omg69 because they get the news out
faster than you can.
They must be tracking a lot more than just the tweets themselves (which
is expected if they want to monetize the service) as even they say 50M a
day: http://blog.twitter.com/2010/02/measuring-tweets.html.
Not saying your wrong, just they must be doing a lot else with it.
Perhaps their logs of delivery of the tweets are just the tweets
themselves: they are small enough perhaps generating a hash of the
message and saving the hash into a log makes less sense than just going
ahead and saving a copy for each person that is subscribed.Either way
crazy data. Lots of people with more data, but I doubt as much that was
typed by hand :-)
On 04/16/2010 02:22 PM, Stu Hood wrote:
> http://twitter.com/jromeh/status/12295736793
>
> -----Original Message-----
> From: "Mike Gallamore"<mi...@googlemail.com>
> Sent: Friday, April 16, 2010 3:46pm
> To: user@cassandra.apache.org
> Subject: Re: Regarding Cassandra Scalability
>
> Also people with 1M followers tend to have "public" tweets, which means
> really I think it would be the same as subscribing to a RSS feed or
> whatever. You aren't getting a local copy because you will "always" have
> access to the tweet as will everyone else. Also tweets don't change
> AFAIK so no point in having redundant copies.
> On 04/16/2010 01:42 PM, Peter Chang wrote:
>
>> Yeah. I wasn't sure if Cassandra was optimized for binary data
>> especially since any site of that size will use a CDN. Interesting
>> read though.
>>
>> I think 1K per tweet is off by an order of magnitude considering they
>> only allow 140 characters. Regardless the number of users with> 1MM
>> is probably a handful. Also im guessing they purge data after a
>> certain window (like 30 days for example).
>>
>> Sent from my iPhone
>>
>>
>> On Apr 16, 2010, at 12:02 PM, gabriele renzi<rf...@gmail.com> wrote:
>>
>>
>>
>>> On Fri, Apr 16, 2010 at 6:41 PM, Peter Chang<pe...@gmail.com>
>>> wrote:
>>>
>>>
>>>> FB also does pics and movies so 1MB is way off depending on where
>>>> they
>>>> manage such binary data.
>>>>
>>>>
>>> apparently not in cassandra http://www.facebook.com/note.php?note_id=76191543919
>>>
>>>
>>>
>>>> I do agree that 1MB of text alone is a lot of text
>>>> which is more relevant in the case of Twitter. The only large thing
>>>> you
>>>> leave out is denormalization. Every tweet you write is likely
>>>> denormalized
>>>> across your followers to allow for quick read access.
>>>>
>>>>
>>> .. but considering many users have _millions_ of followers, this may
>>> be quite a bit more data. Assuming 1k per tweet, this would mean one
>>> from @aplusk (4.7M followers) would take more than 4 gigabytes of
>>> data. Assuming ten tweets a day, in one month he'd produce one TB.
>>>
>>> I'd say they only store references (increasing number lists can also
>>> be encoded very cleverly), or in some other way I'm not smart enough
>>> to think of.
>>>
>>>
>
>
>
Re: Regarding Cassandra Scalability
Posted by Stu Hood <st...@rackspace.com>.
http://twitter.com/jromeh/status/12295736793
-----Original Message-----
From: "Mike Gallamore" <mi...@googlemail.com>
Sent: Friday, April 16, 2010 3:46pm
To: user@cassandra.apache.org
Subject: Re: Regarding Cassandra Scalability
Also people with 1M followers tend to have "public" tweets, which means
really I think it would be the same as subscribing to a RSS feed or
whatever. You aren't getting a local copy because you will "always" have
access to the tweet as will everyone else. Also tweets don't change
AFAIK so no point in having redundant copies.
On 04/16/2010 01:42 PM, Peter Chang wrote:
> Yeah. I wasn't sure if Cassandra was optimized for binary data
> especially since any site of that size will use a CDN. Interesting
> read though.
>
> I think 1K per tweet is off by an order of magnitude considering they
> only allow 140 characters. Regardless the number of users with> 1MM
> is probably a handful. Also im guessing they purge data after a
> certain window (like 30 days for example).
>
> Sent from my iPhone
>
>
> On Apr 16, 2010, at 12:02 PM, gabriele renzi<rf...@gmail.com> wrote:
>
>
>> On Fri, Apr 16, 2010 at 6:41 PM, Peter Chang<pe...@gmail.com>
>> wrote:
>>
>>> FB also does pics and movies so 1MB is way off depending on where
>>> they
>>> manage such binary data.
>>>
>> apparently not in cassandra http://www.facebook.com/note.php?note_id=76191543919
>>
>>
>>> I do agree that 1MB of text alone is a lot of text
>>> which is more relevant in the case of Twitter. The only large thing
>>> you
>>> leave out is denormalization. Every tweet you write is likely
>>> denormalized
>>> across your followers to allow for quick read access.
>>>
>> .. but considering many users have _millions_ of followers, this may
>> be quite a bit more data. Assuming 1k per tweet, this would mean one
>> from @aplusk (4.7M followers) would take more than 4 gigabytes of
>> data. Assuming ten tweets a day, in one month he'd produce one TB.
>>
>> I'd say they only store references (increasing number lists can also
>> be encoded very cleverly), or in some other way I'm not smart enough
>> to think of.
>>
Re: Regarding Cassandra Scalability
Posted by Mike Gallamore <mi...@googlemail.com>.
Also people with 1M followers tend to have "public" tweets, which means
really I think it would be the same as subscribing to a RSS feed or
whatever. You aren't getting a local copy because you will "always" have
access to the tweet as will everyone else. Also tweets don't change
AFAIK so no point in having redundant copies.
On 04/16/2010 01:42 PM, Peter Chang wrote:
> Yeah. I wasn't sure if Cassandra was optimized for binary data
> especially since any site of that size will use a CDN. Interesting
> read though.
>
> I think 1K per tweet is off by an order of magnitude considering they
> only allow 140 characters. Regardless the number of users with> 1MM
> is probably a handful. Also im guessing they purge data after a
> certain window (like 30 days for example).
>
> Sent from my iPhone
>
>
> On Apr 16, 2010, at 12:02 PM, gabriele renzi<rf...@gmail.com> wrote:
>
>
>> On Fri, Apr 16, 2010 at 6:41 PM, Peter Chang<pe...@gmail.com>
>> wrote:
>>
>>> FB also does pics and movies so 1MB is way off depending on where
>>> they
>>> manage such binary data.
>>>
>> apparently not in cassandra http://www.facebook.com/note.php?note_id=76191543919
>>
>>
>>> I do agree that 1MB of text alone is a lot of text
>>> which is more relevant in the case of Twitter. The only large thing
>>> you
>>> leave out is denormalization. Every tweet you write is likely
>>> denormalized
>>> across your followers to allow for quick read access.
>>>
>> .. but considering many users have _millions_ of followers, this may
>> be quite a bit more data. Assuming 1k per tweet, this would mean one
>> from @aplusk (4.7M followers) would take more than 4 gigabytes of
>> data. Assuming ten tweets a day, in one month he'd produce one TB.
>>
>> I'd say they only store references (increasing number lists can also
>> be encoded very cleverly), or in some other way I'm not smart enough
>> to think of.
>>
Re: Regarding Cassandra Scalability
Posted by Peter Chang <pe...@gmail.com>.
Yeah. I wasn't sure if Cassandra was optimized for binary data
especially since any site of that size will use a CDN. Interesting
read though.
I think 1K per tweet is off by an order of magnitude considering they
only allow 140 characters. Regardless the number of users with > 1MM
is probably a handful. Also im guessing they purge data after a
certain window (like 30 days for example).
Sent from my iPhone
On Apr 16, 2010, at 12:02 PM, gabriele renzi <rf...@gmail.com> wrote:
> On Fri, Apr 16, 2010 at 6:41 PM, Peter Chang <pe...@gmail.com>
> wrote:
>> FB also does pics and movies so 1MB is way off depending on where
>> they
>> manage such binary data.
>
> apparently not in cassandra http://www.facebook.com/note.php?note_id=76191543919
>
>> I do agree that 1MB of text alone is a lot of text
>> which is more relevant in the case of Twitter. The only large thing
>> you
>> leave out is denormalization. Every tweet you write is likely
>> denormalized
>> across your followers to allow for quick read access.
>
> .. but considering many users have _millions_ of followers, this may
> be quite a bit more data. Assuming 1k per tweet, this would mean one
> from @aplusk (4.7M followers) would take more than 4 gigabytes of
> data. Assuming ten tweets a day, in one month he'd produce one TB.
>
> I'd say they only store references (increasing number lists can also
> be encoded very cleverly), or in some other way I'm not smart enough
> to think of.
Re: Regarding Cassandra Scalability
Posted by gabriele renzi <rf...@gmail.com>.
On Fri, Apr 16, 2010 at 6:41 PM, Peter Chang <pe...@gmail.com> wrote:
> FB also does pics and movies so 1MB is way off depending on where they
> manage such binary data.
apparently not in cassandra http://www.facebook.com/note.php?note_id=76191543919
>I do agree that 1MB of text alone is a lot of text
> which is more relevant in the case of Twitter. The only large thing you
> leave out is denormalization. Every tweet you write is likely denormalized
> across your followers to allow for quick read access.
.. but considering many users have _millions_ of followers, this may
be quite a bit more data. Assuming 1k per tweet, this would mean one
from @aplusk (4.7M followers) would take more than 4 gigabytes of
data. Assuming ten tweets a day, in one month he'd produce one TB.
I'd say they only store references (increasing number lists can also
be encoded very cleverly), or in some other way I'm not smart enough
to think of.
Re: Regarding Cassandra Scalability
Posted by Peter Chang <pe...@gmail.com>.
FB also does pics and movies so 1MB is way off depending on where they
manage such binary data. I do agree that 1MB of text alone is a lot of text
which is more relevant in the case of Twitter. The only large thing you
leave out is denormalization. Every tweet you write is likely denormalized
across your followers to allow for quick read access.
Sent from my iPhone
On Apr 16, 2010, at 10:17 AM, Mike Gallamore <
mike.e.gallamore@googlemail.com> wrote:
On 04/16/2010 01:38 AM, dir dir wrote:
I hear Facebook.com and tweeter.com using cassandra database. In my opinion
Facebook and
tweeter have hundreds TB data. because their user reach hundreds million
people.
I think you might be forgetting just how tiny tweets are. The last numbers I
heard tweeter gets 55,000,000 messages a day. They've been around for
roughly 4 years. Even assuming they always had that number of messages
(which isn't the case) that still would only be roughly 11TB of data if
everyone sent the maximum tweet length. Sure add a bit to each message for a
time stamp and the user that posted it but still I'd be surprised if every
tweet including meta data was much more than 20TB.
Similarly with Facebook. I think it is the friend list search that they
really did it with. Regardless how much text is on your Facebook page? Maybe
1MB if you are a very very active user. The images I wouldn't think they
would load directly into Cassandra but I could be wrong, I would suspect
that they would pull an old database trick and have filesystem store the
images and the "database" just stores the path to it.
There could be a lot of other data floating around some of which might be in
Cassandra but I don't know. Just the core data that the sites have mentioned
that they use Cassandra for I think is probably in the very low 10's of TB.
Lastly sites like Facebook and Tweeter count hundreds of millions of users
but a lot of them are people that sign in, send a few tweets or connect to a
few friends and then don't use the site again. When the company needs to
make themselves look valuable they count every single person that ever
logged in, even if they only did it once or haven't used the site for years.
They want to sell large numbers because that is what advertisers/potential
acquirers to base the price on those large numbers.
Dir.
On Fri, Apr 16, 2010 at 1:28 PM, Linton N <ga...@gmail.com>wrote:
> hi ,
> I am working for the past 1 year with hadoop, but quite new to
> cassandra, I would like to get clarified few things regarding the
> scalability of Cassandra. Can it scall up to TB of data ?
>
> Please provide me some links regarding this..
>
>
> --
> --
> With Love
> Lin N
>
Re: Regarding Cassandra Scalability
Posted by Mason Hale <ma...@onespot.com>.
On Sat, Apr 17, 2010 at 10:50 AM, dir dir <si...@gmail.com> wrote:
> Hi Mason,
>
> Honestly, I am beginner user in Cassandra. I rather confused to follow
> this database. I ask to the forum about the reason twitter.com to use
> Cassandra
> because I want to know the basic reason why we choose Cassandra instead of
> RDBMS or OODBMS. Twitter.com is only a study case.
>
> Simple, for example I am a student and you are my lecturer. now I am asking
> you
> when we shall use Cassandra? In what situation, we better use Cassandra
> rather than
> RDBMS or OODBMS? Can I use Cassandra to develop Business Information System
> Software such as Accounting Information System??
>
>
Hello Dir --
I'm sorry but answering your question is just not that simple.
The best answer I can give is that one way or another it is likely possible
to build any type of system you can imagine using Cassandra. That said,
there definitely are classes of problems that are going to be handled better
or more easily by a system other than Cassandra. It really depends on what
your needs are. No one is claiming Cassandra is a one-size-fits-all solution
to all data management problems.
If extreme high-availability, support for massive amounts of data, and
scalable write performance are very important to you, then Cassandra is
likely a good fit. If on the other hand you can tolerate some occasional
downtime or if all your data fits comfortably on one machine, then Cassandra
could very well be overkill for your problem. You have to analyze your own
needs and compare those against the available options.
Cassandra is very new, as are all the proverbial "nosql" options that are
the rage these days. You won't find books or training courses covering
Cassandra just yet. The development tool and application framework support
is going to be nascent to non-existent for now. It's an exciting new world
with a lot of innovation and opportunity, but also with a lot of bugs, a
long list of features yet-to-be-implemented, and a lot of people figuring
things out as they go along. So your own tolerance for risk and your
willingness and ability to climb a sometimes steep learning curve should
also factor into your decision.
My advice, if you *think* Cassandra may be a good option for your
application, just try it. Build a prototype and see how that goes.
Mason
Re: Regarding Cassandra Scalability
Posted by Stu Hood <st...@rackspace.com>.
When I made that statement, we were expecting to have fixed the issue within a month or two... the fix has been dragged out significantly due to some disagreements around implementation, but it is a problem that will still be fixed in the near future.
-----Original Message-----
From: "Mason Hale" <ma...@onespot.com>
Sent: Sunday, April 18, 2010 8:53am
To: user@cassandra.apache.org
Subject: Re: Regarding Cassandra Scalability
On Sun, Apr 18, 2010 at 8:26 AM, Brandon Williams <dr...@gmail.com> wrote:
>
> On Sun, Apr 18, 2010 at 8:00 AM, Mason Hale <ma...@onespot.com> wrote:
>
>> This is a statement I wish I had run across sooner. Our first
>> implementation (which we're changing now) included some very big rows. We
>> ran into trouble with compaction and during hinted hand-off operations
>> (which also deals with data a full row at a time) because these rows would
>> not fit into available memory.
>>
>> I think until there are not these lurking gotcha spots like compaction and
>> hinted hand-off, where a full row must fit in memory, we should not be
>> making misleading statements like "Cassandra has the advantage of a more
>> advanced datamodel, allowing for a single row to contain billions of
>> column/value pairs: enough to fill a machine." (from:
>> http://gigaom.com/2010/03/11/digg-cassandara/ ,
>> http://spyced.blogspot.com/2010/03/cassandra-in-action.html). A statement
>> like that should have some caveats, otherwise it reads as an endorsement, a
>> suggestion even, to build a data model with massively wide rows. In
>> practice, it is not feasible to have billions of columns in a single row
>> because it will lead to problems with compaction and hinted hand-off, maybe
>> elsewhere.
>>
>> Mason
>>
>
> http://wiki.apache.org/cassandra/CassandraLimitations
>
> We aren't hiding anything from the user who wishes to educate themselves.
>
> -Brandon
>
>
I didn't mean to imply anyone was hiding information. I was pointing out
that there is conflicting information floating about. And if someone doesn't
read and internalize the entire wiki, they may not realize the quote they
read from Stu Hood about Cassandra describes something that is actually not
a good idea in practice.
In my case, I believe I first read the CassandraLimitations page *after* I
ran into the problem with rows not fitting into memory, and it was after
reading statements about storing billions of columns in a single row. Or
maybe I read the CassandraLimitations page earlier but everything was so new
at that point that it didn't occur to me that the information there negated
the information about storing billions of columns per row. Or maybe I read
both pieces of information, but the one I remembered was the one about
storing billions of columns per row. I don't recall exactly.
My point is -- conflicting information is bad. It relies on the consumer of
that information finding both the "correct" and "incorrect" information and
then accurately reconciling the two. There are many ways that won't work out
well. The best solution is to not have the conflicting information in the
first place.
I'm just offering up my experience as a lesson learned. It is an example of
where I think the Cassandra community can do a better job of communicating.
I just know I had a bit of a bumpy ride because of this issue, and so I'm
pointing it out constructively in hopes that someone after me will not stub
their toe on the same issue.
Mason
Re: Regarding Cassandra Scalability
Posted by Mason Hale <ma...@onespot.com>.
On Sun, Apr 18, 2010 at 8:26 AM, Brandon Williams <dr...@gmail.com> wrote:
>
> On Sun, Apr 18, 2010 at 8:00 AM, Mason Hale <ma...@onespot.com> wrote:
>
>> This is a statement I wish I had run across sooner. Our first
>> implementation (which we're changing now) included some very big rows. We
>> ran into trouble with compaction and during hinted hand-off operations
>> (which also deals with data a full row at a time) because these rows would
>> not fit into available memory.
>>
>> I think until there are not these lurking gotcha spots like compaction and
>> hinted hand-off, where a full row must fit in memory, we should not be
>> making misleading statements like "Cassandra has the advantage of a more
>> advanced datamodel, allowing for a single row to contain billions of
>> column/value pairs: enough to fill a machine." (from:
>> http://gigaom.com/2010/03/11/digg-cassandara/ ,
>> http://spyced.blogspot.com/2010/03/cassandra-in-action.html). A statement
>> like that should have some caveats, otherwise it reads as an endorsement, a
>> suggestion even, to build a data model with massively wide rows. In
>> practice, it is not feasible to have billions of columns in a single row
>> because it will lead to problems with compaction and hinted hand-off, maybe
>> elsewhere.
>>
>> Mason
>>
>
> http://wiki.apache.org/cassandra/CassandraLimitations
>
> We aren't hiding anything from the user who wishes to educate themselves.
>
> -Brandon
>
>
I didn't mean to imply anyone was hiding information. I was pointing out
that there is conflicting information floating about. And if someone doesn't
read and internalize the entire wiki, they may not realize the quote they
read from Stu Hood about Cassandra describes something that is actually not
a good idea in practice.
In my case, I believe I first read the CassandraLimitations page *after* I
ran into the problem with rows not fitting into memory, and it was after
reading statements about storing billions of columns in a single row. Or
maybe I read the CassandraLimitations page earlier but everything was so new
at that point that it didn't occur to me that the information there negated
the information about storing billions of columns per row. Or maybe I read
both pieces of information, but the one I remembered was the one about
storing billions of columns per row. I don't recall exactly.
My point is -- conflicting information is bad. It relies on the consumer of
that information finding both the "correct" and "incorrect" information and
then accurately reconciling the two. There are many ways that won't work out
well. The best solution is to not have the conflicting information in the
first place.
I'm just offering up my experience as a lesson learned. It is an example of
where I think the Cassandra community can do a better job of communicating.
I just know I had a bit of a bumpy ride because of this issue, and so I'm
pointing it out constructively in hopes that someone after me will not stub
their toe on the same issue.
Mason
Re: Regarding Cassandra Scalability
Posted by Brandon Williams <dr...@gmail.com>.
On Sun, Apr 18, 2010 at 8:00 AM, Mason Hale <ma...@onespot.com> wrote:
> This is a statement I wish I had run across sooner. Our first
> implementation (which we're changing now) included some very big rows. We
> ran into trouble with compaction and during hinted hand-off operations
> (which also deals with data a full row at a time) because these rows would
> not fit into available memory.
>
> I think until there are not these lurking gotcha spots like compaction and
> hinted hand-off, where a full row must fit in memory, we should not be
> making misleading statements like "Cassandra has the advantage of a more
> advanced datamodel, allowing for a single row to contain billions of
> column/value pairs: enough to fill a machine." (from:
> http://gigaom.com/2010/03/11/digg-cassandara/ ,
> http://spyced.blogspot.com/2010/03/cassandra-in-action.html). A statement
> like that should have some caveats, otherwise it reads as an endorsement, a
> suggestion even, to build a data model with massively wide rows. In
> practice, it is not feasible to have billions of columns in a single row
> because it will lead to problems with compaction and hinted hand-off, maybe
> elsewhere.
>
> Mason
>
http://wiki.apache.org/cassandra/CassandraLimitations
We aren't hiding anything from the user who wishes to educate themselves.
-Brandon
Re: Regarding Cassandra Scalability
Posted by Schubert Zhang <zs...@gmail.com>.
Yes, it is ture.
Current cassandra has many limitations or bad implementations, especially on
storage level.
In my opinion, these limitations or bad implementations are just
implementation, not the original intention of design.
And I also want to give a suggestion/advice to the project leaders, we
currently should avoid introducing too many useless/unnecessary, and forcus
on fixing bugs/re-implement some core modules.
On Sun, Apr 18, 2010 at 9:00 PM, Mason Hale <ma...@onespot.com> wrote:
>
>
> On Sun, Apr 18, 2010 at 7:41 AM, Gary Dusbabek <gd...@gmail.com>wrote:
>
>> On Sat, Apr 17, 2010 at 10:50, dir dir <si...@gmail.com> wrote:
>> >
>> > What problems can’t it solve?
>> >
>> > No flexible indices
>> > No querying on non PK values
>> > Not good for binary data (>64mb) unless you chunck
>> > Row contents must fit in available memory
>> >
>> > Gary Dusbabek say: Row contents must fit in available memory. Honestly I
>> do
>> > not understand
>> > the meaning from that statement. Thank you.
>> >
>> > Dir.
>> >
>>
>> The main reason is that the compaction operation (removing deleted
>> values) currently requires that an entire row be read into memory.
>>
>> Gary Dusbabek
>>
>
>
> This is a statement I wish I had run across sooner. Our first
> implementation (which we're changing now) included some very big rows. We
> ran into trouble with compaction and during hinted hand-off operations
> (which also deals with data a full row at a time) because these rows would
> not fit into available memory.
>
> I think until there are not these lurking gotcha spots like compaction and
> hinted hand-off, where a full row must fit in memory, we should not be
> making misleading statements like "Cassandra has the advantage of a more
> advanced datamodel, allowing for a single row to contain billions of
> column/value pairs: enough to fill a machine." (from:
> http://gigaom.com/2010/03/11/digg-cassandara/ ,
> http://spyced.blogspot.com/2010/03/cassandra-in-action.html). A statement
> like that should have some caveats, otherwise it reads as an endorsement, a
> suggestion even, to build a data model with massively wide rows. In
> practice, it is not feasible to have billions of columns in a single row
> because it will lead to problems with compaction and hinted hand-off, maybe
> elsewhere.
>
> Mason
>
>
>
Re: Regarding Cassandra Scalability
Posted by Mason Hale <ma...@onespot.com>.
On Sun, Apr 18, 2010 at 7:41 AM, Gary Dusbabek <gd...@gmail.com> wrote:
> On Sat, Apr 17, 2010 at 10:50, dir dir <si...@gmail.com> wrote:
> >
> > What problems can’t it solve?
> >
> > No flexible indices
> > No querying on non PK values
> > Not good for binary data (>64mb) unless you chunck
> > Row contents must fit in available memory
> >
> > Gary Dusbabek say: Row contents must fit in available memory. Honestly I
> do
> > not understand
> > the meaning from that statement. Thank you.
> >
> > Dir.
> >
>
> The main reason is that the compaction operation (removing deleted
> values) currently requires that an entire row be read into memory.
>
> Gary Dusbabek
>
This is a statement I wish I had run across sooner. Our first implementation
(which we're changing now) included some very big rows. We ran into trouble
with compaction and during hinted hand-off operations (which also deals with
data a full row at a time) because these rows would not fit into available
memory.
I think until there are not these lurking gotcha spots like compaction and
hinted hand-off, where a full row must fit in memory, we should not be
making misleading statements like "Cassandra has the advantage of a more
advanced datamodel, allowing for a single row to contain billions of
column/value pairs: enough to fill a machine." (from:
http://gigaom.com/2010/03/11/digg-cassandara/ ,
http://spyced.blogspot.com/2010/03/cassandra-in-action.html). A statement
like that should have some caveats, otherwise it reads as an endorsement, a
suggestion even, to build a data model with massively wide rows. In
practice, it is not feasible to have billions of columns in a single row
because it will lead to problems with compaction and hinted hand-off, maybe
elsewhere.
Mason
Re: Regarding Cassandra Scalability
Posted by dir dir <si...@gmail.com>.
Hi Paul,
I do not have any pressure to build software using Cassandra right now.
I am studying and exploring Cassandra now. Hence I have a big curiosity
about Cassandra. Ok I will continue my study and wait better documentation.
Dir.
On Mon, Apr 19, 2010 at 1:44 PM, Paul Prescod <pr...@gmail.com> wrote:
> On Sun, Apr 18, 2010 at 9:14 AM, dir dir <si...@gmail.com> wrote:
> > Hi Gary,
> >
> >>The main reason is that the compaction operation (removing deleted
> >>values) currently requires that an entire row be read into memory.
> >
> > Thank you for your explanation. But I still do not understand what do you
> > mean.
>
> Do you have a pressing need to use Cassandra right now, before version
> 1.0 is even available?
>
> That limitation will go away before 1.0, so you could simply wait and
> not worry about it. Documentation will also be much more complete in
> the future.
>
> Paul Prescod
>
Re: Regarding Cassandra Scalability
Posted by Paul Prescod <pr...@gmail.com>.
On Sun, Apr 18, 2010 at 9:14 AM, dir dir <si...@gmail.com> wrote:
> Hi Gary,
>
>>The main reason is that the compaction operation (removing deleted
>>values) currently requires that an entire row be read into memory.
>
> Thank you for your explanation. But I still do not understand what do you
> mean.
Do you have a pressing need to use Cassandra right now, before version
1.0 is even available?
That limitation will go away before 1.0, so you could simply wait and
not worry about it. Documentation will also be much more complete in
the future.
Paul Prescod
Re: Regarding Cassandra Scalability
Posted by Gary Dusbabek <gd...@gmail.com>.
On Sun, Apr 18, 2010 at 11:14, dir dir <si...@gmail.com> wrote:
> Hi Gary,
>
>>The main reason is that the compaction operation (removing deleted
>>values) currently requires that an entire row be read into memory.
>
> Thank you for your explanation. But I still do not understand what do you
> mean.
>
When you delete a column in cassandra, the data is not really deleted.
Instead a flag is turned on indicating the column is no longer valid
(we call it a 'tombstone'). During compaction the column family is
scanned and the tombstones are truly deleted.
> in my opinion, Actually the row contents must fit in available memory.
> if row contents are not fit in available memory, our software will raise
> exception out of memory. since it is true( "the row contents must fit in
> available memory"),
> then why you said that is a problem which it (Cassandra) cannot solved??
>
It was not correct of me to say it is a problem that cassandra cannot
solve. Memory-efficient compactions will be addressed.
> You say: "compaction operation requires that entire row be read into memory"
> whether this is a problem of "out of memory"?? When we need to perform
> compaction operation?? In what situation we shall perform compaction
> operation??
You will need to address the large rows yourself (consider breaking
them up). You can identify these rows during compaction by setting
RowWarningThresholdInMB in storage-conf.xml. When a big enough row
comes along, it is logged so you can go back later and address the
problem.
Regards,
Gary.
> Thank You.
>
> Dir.
>
Re: Regarding Cassandra Scalability
Posted by dir dir <si...@gmail.com>.
Hi Gary,
>The main reason is that the compaction operation (removing deleted
>values) currently requires that an entire row be read into memory.
Thank you for your explanation. But I still do not understand what do you
mean.
in my opinion, Actually the row contents must fit in available memory.
if row contents are not fit in available memory, our software will raise
exception out of memory. since it is true( "the row contents must fit in
available memory"),
then why you said that is a problem which it (Cassandra) cannot solved??
You say: "compaction operation requires that entire row be read into memory"
whether this is a problem of "out of memory"?? When we need to perform
compaction operation?? In what situation we shall perform compaction
operation??
Thank You.
Dir.
On Sun, Apr 18, 2010 at 7:41 PM, Gary Dusbabek <gd...@gmail.com> wrote:
> On Sat, Apr 17, 2010 at 10:50, dir dir <si...@gmail.com> wrote:
> >
> > What problems can’t it solve?
> >
> > No flexible indices
> > No querying on non PK values
> > Not good for binary data (>64mb) unless you chunck
> > Row contents must fit in available memory
> >
> > Gary Dusbabek say: Row contents must fit in available memory. Honestly I
> do
> > not understand
> > the meaning from that statement. Thank you.
> >
> > Dir.
> >
>
> The main reason is that the compaction operation (removing deleted
> values) currently requires that an entire row be read into memory.
>
> Gary Dusbabek
>
Re: Regarding Cassandra Scalability
Posted by Gary Dusbabek <gd...@gmail.com>.
On Sat, Apr 17, 2010 at 10:50, dir dir <si...@gmail.com> wrote:
>
> What problems can’t it solve?
>
> No flexible indices
> No querying on non PK values
> Not good for binary data (>64mb) unless you chunck
> Row contents must fit in available memory
>
> Gary Dusbabek say: Row contents must fit in available memory. Honestly I do
> not understand
> the meaning from that statement. Thank you.
>
> Dir.
>
The main reason is that the compaction operation (removing deleted
values) currently requires that an entire row be read into memory.
Gary Dusbabek
Re: Regarding Cassandra Scalability
Posted by dir dir <si...@gmail.com>.
Hi Mason,
Honestly, I am beginner user in Cassandra. I rather confused to follow
this database. I ask to the forum about the reason twitter.com to use
Cassandra
because I want to know the basic reason why we choose Cassandra instead of
RDBMS or OODBMS. Twitter.com is only a study case.
Simple, for example I am a student and you are my lecturer. now I am asking
you
when we shall use Cassandra? In what situation, we better use Cassandra
rather than
RDBMS or OODBMS? Can I use Cassandra to develop Business Information System
Software such as Accounting Information System??
I have already read the article on
http://nosql.mypopescu.com/post/517791994/presentation-gary-dusbabek-rackspace-on-cassandra
If I see the Cassandra's Data Model, it seem Cassandra can be applied in
many
software application, like RDBMS. It is true or not?? I also have a question
related with
the presentation on the website.
What problems can’t it solve?
- No flexible indices
- No querying on non PK values
- Not good for binary data (>64mb) unless you chunck
- Row contents must fit in available memory
Gary Dusbabek say: Row contents must fit in available memory. Honestly I do
not understand
the meaning from that statement. Thank you.
Dir.
On Sat, Apr 17, 2010 at 8:46 PM, Mason Hale <ma...@onespot.com> wrote:
> On Saturday, April 17, 2010, dir dir
> > I want to ask to the advance user or experience software developer in
> this forum,
> > why tweeter.com choose Cassandra? would you tell me the reason behind
> > the decision of tweeter.com (from the feature and technical aspect)??
> Why tweeter.com did not
> > use Oracle 11g or Db4o for example? (please omit the oracle 11g fee
> license because
> > we shall discuss from the feature and technical aspect). Thank you.
> >
> >
>
> For clarification, I believe the site you are referring to is
> twitter.com (not tweeter.com). Only people at Twitter who made this
> decision can really explain the rationale behind it.
>
> This link should shed some light on the subject:
> http://engineering.twitter.com/2010/02/link-cassandra-at-twitter.html
>
> Mason
>
>
>
>
> --
>
> Mason Hale
> http://www.onespot.com
> direct +1 800.618.0768 ext 701
>
Re: Regarding Cassandra Scalability
Posted by Mason Hale <ma...@onespot.com>.
On Saturday, April 17, 2010, dir dir
> I want to ask to the advance user or experience software developer in this forum,
> why tweeter.com choose Cassandra? would you tell me the reason behind
> the decision of tweeter.com (from the feature and technical aspect)?? Why tweeter.com did not
> use Oracle 11g or Db4o for example? (please omit the oracle 11g fee license because
> we shall discuss from the feature and technical aspect). Thank you.
>
>
For clarification, I believe the site you are referring to is
twitter.com (not tweeter.com). Only people at Twitter who made this
decision can really explain the rationale behind it.
This link should shed some light on the subject:
http://engineering.twitter.com/2010/02/link-cassandra-at-twitter.html
Mason
--
Mason Hale
http://www.onespot.com
direct +1 800.618.0768 ext 701
Re: Regarding Cassandra Scalability
Posted by dir dir <si...@gmail.com>.
>I think you might be forgetting just how tiny tweets are. The last numbers
I heard tweeter gets 55,000,000 messages a day. They've been around for
>roughly 4 years.
I read a news in the internet, in the beginning tweeter using RDBMS MySQL
until tweeter
reach amount of tweet 1 million per day. Since the user of
tweeter.comexploding,
tweeter.com decided using Cassandra database and shutdown MySQL database.
I want to ask to the advance user or experience software developer in this
forum,
why tweeter.com choose Cassandra? would you tell me the reason behind
the decision of tweeter.com (from the feature and technical aspect)?? Why
tweeter.com did not
use Oracle 11g or Db4o for example? (please omit the oracle 11g fee license
because
we shall discuss from the feature and technical aspect). Thank you.
Dir.
On Fri, Apr 16, 2010 at 11:17 PM, Mike Gallamore <
mike.e.gallamore@googlemail.com> wrote:
> On 04/16/2010 01:38 AM, dir dir wrote:
>
> I hear Facebook.com and tweeter.com using cassandra database. In my
> opinion Facebook and
> tweeter have hundreds TB data. because their user reach hundreds million
> people.
>
> I think you might be forgetting just how tiny tweets are. The last numbers
> I heard tweeter gets 55,000,000 messages a day. They've been around for
> roughly 4 years. Even assuming they always had that number of messages
> (which isn't the case) that still would only be roughly 11TB of data if
> everyone sent the maximum tweet length. Sure add a bit to each message for a
> time stamp and the user that posted it but still I'd be surprised if every
> tweet including meta data was much more than 20TB.
>
> Similarly with Facebook. I think it is the friend list search that they
> really did it with. Regardless how much text is on your Facebook page? Maybe
> 1MB if you are a very very active user. The images I wouldn't think they
> would load directly into Cassandra but I could be wrong, I would suspect
> that they would pull an old database trick and have filesystem store the
> images and the "database" just stores the path to it.
>
> There could be a lot of other data floating around some of which might be
> in Cassandra but I don't know. Just the core data that the sites have
> mentioned that they use Cassandra for I think is probably in the very low
> 10's of TB.
>
> Lastly sites like Facebook and Tweeter count hundreds of millions of users
> but a lot of them are people that sign in, send a few tweets or connect to a
> few friends and then don't use the site again. When the company needs to
> make themselves look valuable they count every single person that ever
> logged in, even if they only did it once or haven't used the site for years.
> They want to sell large numbers because that is what advertisers/potential
> acquirers to base the price on those large numbers.
>
>
> Dir.
>
>
> On Fri, Apr 16, 2010 at 1:28 PM, Linton N <ga...@gmail.com>wrote:
>
>> hi ,
>> I am working for the past 1 year with hadoop, but quite new to
>> cassandra, I would like to get clarified few things regarding the
>> scalability of Cassandra. Can it scall up to TB of data ?
>>
>> Please provide me some links regarding this..
>>
>>
>> --
>> --
>> With Love
>> Lin N
>>
>
>
>
Re: Regarding Cassandra Scalability
Posted by Tatu Saloranta <ts...@gmail.com>.
On Fri, Apr 16, 2010 at 9:17 AM, Mike Gallamore
<mi...@googlemail.com> wrote:
> On 04/16/2010 01:38 AM, dir dir wrote:
>
> I hear Facebook.com and tweeter.com using cassandra database. In my opinion
> Facebook and
> tweeter have hundreds TB data. because their user reach hundreds million
> people.
>
> I think you might be forgetting just how tiny tweets are. The last numbers I
> heard tweeter gets 55,000,000 messages a day. They've been around for
> roughly 4 years. Even assuming they always had that number of messages
> (which isn't the case) that still would only be roughly 11TB of data if
> everyone sent the maximum tweet length. Sure add a bit to each message for a
> time stamp and the user that posted it but still I'd be surprised if every
> tweet including meta data was much more than 20TB.
While valid points, I think there are separate issues wrt searching
and indexing, where number of entries is more relevant. I mean,
storing and accessing big BLOBs is not trivial, but in many ways it is
less problematic than craploads of smaller entries. This is probably
more so for FB and LinkedIn, with more graph-oriented challenges. But
it's not just (or even mainly) about raw storage but all the slicing
and dicing that makes things challenging.
-+ Tatu +-
Re: Regarding Cassandra Scalability
Posted by Mike Gallamore <mi...@googlemail.com>.
On 04/16/2010 01:38 AM, dir dir wrote:
> I hear Facebook.com and tweeter.com <http://tweeter.com> using
> cassandra database. In my opinion Facebook and
> tweeter have hundreds TB data. because their user reach hundreds
> million people.
I think you might be forgetting just how tiny tweets are. The last
numbers I heard tweeter gets 55,000,000 messages a day. They've been
around for roughly 4 years. Even assuming they always had that number of
messages (which isn't the case) that still would only be roughly 11TB of
data if everyone sent the maximum tweet length. Sure add a bit to each
message for a time stamp and the user that posted it but still I'd be
surprised if every tweet including meta data was much more than 20TB.
Similarly with Facebook. I think it is the friend list search that they
really did it with. Regardless how much text is on your Facebook page?
Maybe 1MB if you are a very very active user. The images I wouldn't
think they would load directly into Cassandra but I could be wrong, I
would suspect that they would pull an old database trick and have
filesystem store the images and the "database" just stores the path to it.
There could be a lot of other data floating around some of which might
be in Cassandra but I don't know. Just the core data that the sites have
mentioned that they use Cassandra for I think is probably in the very
low 10's of TB.
Lastly sites like Facebook and Tweeter count hundreds of millions of
users but a lot of them are people that sign in, send a few tweets or
connect to a few friends and then don't use the site again. When the
company needs to make themselves look valuable they count every single
person that ever logged in, even if they only did it once or haven't
used the site for years. They want to sell large numbers because that is
what advertisers/potential acquirers to base the price on those large
numbers.
>
> Dir.
>
>
> On Fri, Apr 16, 2010 at 1:28 PM, Linton N
> <gabrialmarialinton@gmail.com <ma...@gmail.com>>
> wrote:
>
> hi ,
> I am working for the past 1 year with hadoop, but quite
> new to cassandra, I would like to get clarified few things
> regarding the scalability of Cassandra. Can it scall up to TB of
> data ?
>
> Please provide me some links regarding this..
>
>
> --
> --
> With Love
> Lin N
>
>
Re: Regarding Cassandra Scalability
Posted by dir dir <si...@gmail.com>.
I hear Facebook.com and tweeter.com using cassandra database. In my opinion
Facebook and
tweeter have hundreds TB data. because their user reach hundreds million
people.
Dir.
On Fri, Apr 16, 2010 at 1:28 PM, Linton N <ga...@gmail.com>wrote:
> hi ,
> I am working for the past 1 year with hadoop, but quite new to
> cassandra, I would like to get clarified few things regarding the
> scalability of Cassandra. Can it scall up to TB of data ?
>
> Please provide me some links regarding this..
>
>
> --
> --
> With Love
> Lin N
>
Re: Regarding Cassandra Scalability
Posted by Linton N <ga...@gmail.com>.
Thank you very much. sorry for the trouble. I could have done in
myself........
On Fri, Apr 16, 2010 at 1:29 PM, Paul Prescod <pr...@gmail.com> wrote:
> http://www.google.ca/search?hl=en&q=cassandra+terabyte
>
> On Thu, Apr 15, 2010 at 11:28 PM, Linton N <ga...@gmail.com>
> wrote:
> > hi ,
> > I am working for the past 1 year with hadoop, but quite new to
> > cassandra, I would like to get clarified few things regarding the
> > scalability of Cassandra. Can it scall up to TB of data ?
> >
> > Please provide me some links regarding this..
> >
> >
> > --
> > --
> > With Love
> > Lin N
> >
>
--
--
With Love
Lin N
Re: Regarding Cassandra Scalability
Posted by Paul Prescod <pr...@gmail.com>.
http://www.google.ca/search?hl=en&q=cassandra+terabyte
On Thu, Apr 15, 2010 at 11:28 PM, Linton N <ga...@gmail.com> wrote:
> hi ,
> I am working for the past 1 year with hadoop, but quite new to
> cassandra, I would like to get clarified few things regarding the
> scalability of Cassandra. Can it scall up to TB of data ?
>
> Please provide me some links regarding this..
>
>
> --
> --
> With Love
> Lin N
>