You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Simon Reavely <si...@gmail.com> on 2010/11/21 17:32:53 UTC
Facebook messaging and choice of HBase over Cassandra - what can we learn?
(Posting this to both user + dev lists)
I was reviewing the blog post on the facebook engineering blog from nov
15th
http://www.facebook.com/note.php?note_id=454991608919#
<http://www.facebook.com/note.php?note_id=454991608919#>
The Underlying Technology of Messages
by Kannan Muthukkaruppan <http://www.facebook.com/Kannan>
As a cassandra user I think the key sentence for this community is:
"We found Cassandra's eventual consistency model to be a difficult pattern
to reconcile for our new Messages infrastructure."
I think it would be useful to find out more about this statement from Kannan
and the facebook team. Does anyone have any contacts in the Facebook team?
My goal here is to understand usage patterns and whether or not the
Cassandra community can learn from this decision; maybe even understand
whether the Cassandra roadmap should be influenced by this decision to
address a target user base. Of course we might also conclude that its just
"not a Cassandra use-case"!
Cheers,
Simon
--
Simon Reavely
simon.reavely@gmail.com
Re: Facebook messaging and choice of HBase over Cassandra - what can we learn?
Posted by aaron morton <aa...@thelastpickle.com>.
Cassandra can work in a consistent way, see some of this discussion and the Consistency section here http://wiki.apache.org/cassandra/ArchitectureOverview
If you always read and write with CL.Quorum (or the other way discussed) you will have consistency. Even if some of the replicas are temporarily inconsistent, or off line or whatever. Your reads will be consistent, i.e. every client will get the same value or the read will not work. If you want to work at a lower or higher consistency you can.
Eventually all replicas of a value will become consistent.
There are a number of reasons why cassandra may not be a good fit, and I would guess something else would be a problem before the consistency model.
Hope that helps.
Aaron
On 22 Nov 2010, at 21:57, David Boxenhorn wrote:
> It's true that Cassandra has "tunable consistency", but if eventual consistency is not sufficient for most of your use cases, Cassandra becomes much less attractive. Am I wrong?
>
>
>
> On Sun, Nov 21, 2010 at 7:56 PM, Eric Evans <ee...@rackspace.com> wrote:
> On Sun, 2010-11-21 at 11:32 -0500, Simon Reavely wrote:
> > As a cassandra user I think the key sentence for this community is:
> > "We found Cassandra's eventual consistency model to be a difficult
> > pattern to reconcile for our new Messages infrastructure."
>
> In my experience, "we needed strong consistency", in conversations like
> these amounts to hand waving. It's the fastest way to shut down that
> part of the discussion without having said anything at all.
>
> > I think it would be useful to find out more about this statement from
> > Kannan and the facebook team. Does anyone have any contacts in the
> > Facebook team?
>
> Good luck. Facebook is notoriously tight-lipped about such things.
>
> > My goal here is to understand usage patterns and whether or not the
> > Cassandra community can learn from this decision; maybe even
> > understand whether the Cassandra roadmap should be influenced by this
> > decision to address a target user base. Of course we might also
> > conclude that its just "not a Cassandra use-case"!
>
> Understanding is a laudable goal, just try to avoid drawing conclusions
> (and call out others who are).
>
> <rant>
> This is usually the point where a frenzy kicks in and folks assume that
> the Smart Guys at Facebook know something they don't, something that
> would invalidate their decision if they'd only known.
>
> I seriously doubt they've uncovered some Truth that would fundamentally
> alter the reasoning behind *my* decision to use Cassandra, and so I plan
> to continue as I always have. Following relevant research and
> development, collecting experience (my own and others), and applying it
> to the problems I face.
> </rant>
>
> --
> Eric Evans
> eevans@rackspace.com
>
>
Re: Facebook messaging and choice of HBase over Cassandra - what can
we learn?
Posted by David Boxenhorn <da...@lookin2.com>.
It's true that Cassandra has "tunable consistency", but if eventual
consistency is not sufficient for most of your use cases, Cassandra becomes
much less attractive. Am I wrong?
On Sun, Nov 21, 2010 at 7:56 PM, Eric Evans <ee...@rackspace.com> wrote:
> On Sun, 2010-11-21 at 11:32 -0500, Simon Reavely wrote:
> > As a cassandra user I think the key sentence for this community is:
> > "We found Cassandra's eventual consistency model to be a difficult
> > pattern to reconcile for our new Messages infrastructure."
>
> In my experience, "we needed strong consistency", in conversations like
> these amounts to hand waving. It's the fastest way to shut down that
> part of the discussion without having said anything at all.
>
> > I think it would be useful to find out more about this statement from
> > Kannan and the facebook team. Does anyone have any contacts in the
> > Facebook team?
>
> Good luck. Facebook is notoriously tight-lipped about such things.
>
> > My goal here is to understand usage patterns and whether or not the
> > Cassandra community can learn from this decision; maybe even
> > understand whether the Cassandra roadmap should be influenced by this
> > decision to address a target user base. Of course we might also
> > conclude that its just "not a Cassandra use-case"!
>
> Understanding is a laudable goal, just try to avoid drawing conclusions
> (and call out others who are).
>
> <rant>
> This is usually the point where a frenzy kicks in and folks assume that
> the Smart Guys at Facebook know something they don't, something that
> would invalidate their decision if they'd only known.
>
> I seriously doubt they've uncovered some Truth that would fundamentally
> alter the reasoning behind *my* decision to use Cassandra, and so I plan
> to continue as I always have. Following relevant research and
> development, collecting experience (my own and others), and applying it
> to the problems I face.
> </rant>
>
> --
> Eric Evans
> eevans@rackspace.com
>
>
Re: Facebook messaging and choice of HBase over Cassandra - what
can we learn?
Posted by Eric Evans <ee...@rackspace.com>.
On Sun, 2010-11-21 at 11:32 -0500, Simon Reavely wrote:
> As a cassandra user I think the key sentence for this community is:
> "We found Cassandra's eventual consistency model to be a difficult
> pattern to reconcile for our new Messages infrastructure."
In my experience, "we needed strong consistency", in conversations like
these amounts to hand waving. It's the fastest way to shut down that
part of the discussion without having said anything at all.
> I think it would be useful to find out more about this statement from
> Kannan and the facebook team. Does anyone have any contacts in the
> Facebook team?
Good luck. Facebook is notoriously tight-lipped about such things.
> My goal here is to understand usage patterns and whether or not the
> Cassandra community can learn from this decision; maybe even
> understand whether the Cassandra roadmap should be influenced by this
> decision to address a target user base. Of course we might also
> conclude that its just "not a Cassandra use-case"!
Understanding is a laudable goal, just try to avoid drawing conclusions
(and call out others who are).
<rant>
This is usually the point where a frenzy kicks in and folks assume that
the Smart Guys at Facebook know something they don't, something that
would invalidate their decision if they'd only known.
I seriously doubt they've uncovered some Truth that would fundamentally
alter the reasoning behind *my* decision to use Cassandra, and so I plan
to continue as I always have. Following relevant research and
development, collecting experience (my own and others), and applying it
to the problems I face.
</rant>
--
Eric Evans
eevans@rackspace.com
Re: Facebook messaging and choice of HBase over Cassandra - what can
we learn?
Posted by "Richard L. Burton III" <mr...@gmail.com>.
The author of Cassandra is on the Cassandra mailing list. You may want to
ask there. He often replies to e-mails as well.
On Sun, Nov 21, 2010 at 11:32 AM, Simon Reavely <si...@gmail.com>wrote:
> (Posting this to both user + dev lists)
>
> I was reviewing the blog post on the facebook engineering blog from nov
> 15th
> http://www.facebook.com/note.php?note_id=454991608919#
> <http://www.facebook.com/note.php?note_id=454991608919#>
> The Underlying Technology of Messages
> by Kannan Muthukkaruppan <http://www.facebook.com/Kannan>
>
>
>
> As a cassandra user I think the key sentence for this community is:
> "We found Cassandra's eventual consistency model to be a difficult pattern
> to reconcile for our new Messages infrastructure."
>
> I think it would be useful to find out more about this statement from
> Kannan
> and the facebook team. Does anyone have any contacts in the Facebook team?
>
> My goal here is to understand usage patterns and whether or not the
> Cassandra community can learn from this decision; maybe even understand
> whether the Cassandra roadmap should be influenced by this decision to
> address a target user base. Of course we might also conclude that its just
> "not a Cassandra use-case"!
>
> Cheers,
> Simon
> --
> Simon Reavely
> simon.reavely@gmail.com
>
--
-Richard L. Burton III
Re: Facebook messaging and choice of HBase over Cassandra - what can
we learn?
Posted by Dave Viner <da...@gmail.com>.
I don't know the details of operation of HBase, so I can't speak on that
point. But, I do know that Facebook hired Jonathan Grey, former CTO of
Streamy, who is a huge HBase contributor. Streamy ended in Mar 2010 -
although I'm not sure when he went to work for Facebook.
He presented on HBase at the Hadoop conference in October in NYC:
http://mpouttuclarke.wordpress.com/2010/10/18/notes-from-hadoop-world-2010-nyc/
Again, I don't know the chronology (whether he was hired before the decision
to use hbase or after). But I know that Jonathan is a fantastically smart
(and extremely nice) guy and I'm sure he could make HBase bend to his will
at any point.
Dave Viner
On Sun, Nov 21, 2010 at 4:16 PM, Todd Lipcon <to...@lipcon.org> wrote:
> On Sun, Nov 21, 2010 at 2:06 PM, Edward Ribeiro <ed...@gmail.com>wrote:
>
>>
>> Also I believe saying HBASE is consistent is not true. This can happen:
>>> Write to region server. -> Region Server acknowledges client-> write
>>> to WAL -> region server fails = write lost
>>>
>>> I wonder how facebook will reconcile that. :)
>>>
>>
>> Are you sure about that? Client writes to WAL before ack user?
>>
>> According to these posts[1][2], "if writing the record to the WAL fails
>> the whole operation must be considered a failure.", so it would be nonsense
>> acknowledge clients before writing the lifeline. I hope any cloudera guy
>> explain this...
>>
>>
> [only jumping in because info was requested - those who know me know that I
> think Cassandra is a very interesting architecture and a better fit for many
> applications than HBase]
>
> You can operate the commit log in two different modes in HBase. One mode is
> "deferred log flush", where the region server appends but does not sync()
> the commit log to HDFS on every write, but rather on a periodic basis (eg
> once a second). This is similar to the innodb_flush_log_at_trx_commit=2
> option in MySQL for example. This has slightly better performance obviously
> since the writer doesn't need to wait on the commit, but as you noted
> there's a window where a write may be acknowledged but then lost. This is an
> issue of *durability* moreso than consistency.
>
> In the other mode of operation (default in recent versions of HBase) we do
> not acknowledge a write until it has been pushed to the OS buffer on the
> entire pipeline of log replicas. Obviously this is slower, but it results in
> "no lost data" regardless of any machine failures. Additionally, concurrent
> readers do not see written data until these same properties have been
> satisfied. So this mode is 100% consistent and 100% durable. In practice,
> this effects latency significantly since it adds two extra round trips to
> each write, but system throughput is only reduced by 20-30% since the
> commits are pipelined (see HDFS-895 for gory details)
>
> I believe Cassandra has similar tuning options about whether to sync every
> commit to the log or only do so periodically.
>
> If you're interested in learning more, feel free to reference this
> documentation:
> http://hbase.apache.org/docs/r0.89.20100726/acid-semantics.html
>
>
>
>> Besides that, you know that WAL is written to HDFS that takes care of
>> replication and fault tolerance, right? Of course, even so, there's a
>> "window of inconsistency" before the HLog is flushed to disk, but I don't
>> think you can dismiss this as not consistent. At most, you may classify it
>> as "eventual consistent". :)
>>
>> [1] http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html
>> [2]
>> http://www.larsgeorge.com/2010/01/hbase-architecture-101-write-ahead-log.html
>>
>> E. Ribeiro
>>
>>
>
Re: Facebook messaging and choice of HBase over Cassandra - what can
we learn?
Posted by Todd Lipcon <to...@lipcon.org>.
On Sun, Nov 21, 2010 at 6:25 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> On Sun, Nov 21, 2010 at 6:16 PM, Todd Lipcon <to...@lipcon.org> wrote:
> > [only jumping in because info was requested - those who know me know that
> I
> > think Cassandra is a very interesting architecture and a better fit for
> many
> > applications than HBase]
>
> Hey Todd! Good to see you de-lurk!
>
> Howdy :)
> > In the other mode of operation (default in recent versions of HBase) we
> do
> > not acknowledge a write until it has been pushed to the OS buffer on the
> > entire pipeline of log replicas.
>
> You mean "to the disk," I assume?
>
>
Actually just to the OS buffers. So if the entire cluster loses power
simultaneously you will lose data. If you lose just one rack, though, you'll
be OK since the pipeline always spans two racks in a multirack installation.
There are plans to add an API that fully fsync()s the replicas, but
currently there has been little demand - in memory on three replicas is
"safe enough" for most use cases, and of course a lot more performant.
-Todd
Re: Facebook messaging and choice of HBase over Cassandra - what can
we learn?
Posted by Jonathan Ellis <jb...@gmail.com>.
On Sun, Nov 21, 2010 at 6:16 PM, Todd Lipcon <to...@lipcon.org> wrote:
> [only jumping in because info was requested - those who know me know that I
> think Cassandra is a very interesting architecture and a better fit for many
> applications than HBase]
Hey Todd! Good to see you de-lurk!
> In the other mode of operation (default in recent versions of HBase) we do
> not acknowledge a write until it has been pushed to the OS buffer on the
> entire pipeline of log replicas.
You mean "to the disk," I assume?
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: Facebook messaging and choice of HBase over Cassandra - what can
we learn?
Posted by Todd Lipcon <to...@lipcon.org>.
On Sun, Nov 21, 2010 at 2:06 PM, Edward Ribeiro <ed...@gmail.com>wrote:
>
> Also I believe saying HBASE is consistent is not true. This can happen:
>> Write to region server. -> Region Server acknowledges client-> write
>> to WAL -> region server fails = write lost
>>
>> I wonder how facebook will reconcile that. :)
>>
>
> Are you sure about that? Client writes to WAL before ack user?
>
> According to these posts[1][2], "if writing the record to the WAL fails the
> whole operation must be considered a failure.", so it would be nonsense
> acknowledge clients before writing the lifeline. I hope any cloudera guy
> explain this...
>
>
[only jumping in because info was requested - those who know me know that I
think Cassandra is a very interesting architecture and a better fit for many
applications than HBase]
You can operate the commit log in two different modes in HBase. One mode is
"deferred log flush", where the region server appends but does not sync()
the commit log to HDFS on every write, but rather on a periodic basis (eg
once a second). This is similar to the innodb_flush_log_at_trx_commit=2
option in MySQL for example. This has slightly better performance obviously
since the writer doesn't need to wait on the commit, but as you noted
there's a window where a write may be acknowledged but then lost. This is an
issue of *durability* moreso than consistency.
In the other mode of operation (default in recent versions of HBase) we do
not acknowledge a write until it has been pushed to the OS buffer on the
entire pipeline of log replicas. Obviously this is slower, but it results in
"no lost data" regardless of any machine failures. Additionally, concurrent
readers do not see written data until these same properties have been
satisfied. So this mode is 100% consistent and 100% durable. In practice,
this effects latency significantly since it adds two extra round trips to
each write, but system throughput is only reduced by 20-30% since the
commits are pipelined (see HDFS-895 for gory details)
I believe Cassandra has similar tuning options about whether to sync every
commit to the log or only do so periodically.
If you're interested in learning more, feel free to reference this
documentation:
http://hbase.apache.org/docs/r0.89.20100726/acid-semantics.html
> Besides that, you know that WAL is written to HDFS that takes care of
> replication and fault tolerance, right? Of course, even so, there's a
> "window of inconsistency" before the HLog is flushed to disk, but I don't
> think you can dismiss this as not consistent. At most, you may classify it
> as "eventual consistent". :)
>
> [1] http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html
> [2]
> http://www.larsgeorge.com/2010/01/hbase-architecture-101-write-ahead-log.html
>
> E. Ribeiro
>
>
Re: Facebook messaging and choice of HBase over Cassandra - what can
we learn?
Posted by Edward Ribeiro <ed...@gmail.com>.
> Also I believe saying HBASE is consistent is not true. This can happen:
> Write to region server. -> Region Server acknowledges client-> write
> to WAL -> region server fails = write lost
>
> I wonder how facebook will reconcile that. :)
>
Are you sure about that? Client writes to WAL before ack user?
According to these posts[1][2], "if writing the record to the WAL fails the
whole operation must be considered a failure.", so it would be nonsense
acknowledge clients before writing the lifeline. I hope any cloudera guy
explain this...
Besides that, you know that WAL is written to HDFS that takes care of
replication and fault tolerance, right? Of course, even so, there's a
"window of inconsistency" before the HLog is flushed to disk, but I don't
think you can dismiss this as not consistent. At most, you may classify it
as "eventual consistent". :)
[1] http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html
[2]
http://www.larsgeorge.com/2010/01/hbase-architecture-101-write-ahead-log.html
E. Ribeiro
Re: Facebook messaging and choice of HBase over Cassandra - what can we learn?
Posted by Jake Luciani <ja...@gmail.com>.
+1 Ed
On Nov 21, 2010, at 12:13 PM, Edward Capriolo <ed...@gmail.com> wrote:
> On Sun, Nov 21, 2010 at 12:10 PM, André Fiedler
> <fi...@googlemail.com> wrote:
>> Facebook Messaging – HBase Comes of Age
>>
>> http://facility9.com/2010/11/18/facebook-messaging-hbase-comes-of-age
>>
>> 2010/11/21 David Boxenhorn <da...@lookin2.com>
>>>
>>> Eventual consistency is not good enough for instant messaging.
>>>
>>> On Sun, Nov 21, 2010 at 6:32 PM, Simon Reavely <si...@gmail.com>
>>> wrote:
>>>>
>>>> (Posting this to both user + dev lists)
>>>>
>>>> I was reviewing the blog post on the facebook engineering blog from nov
>>>> 15th
>>>> http://www.facebook.com/note.php?note_id=454991608919#
>>>> <http://www.facebook.com/note.php?note_id=454991608919#>
>>>> The Underlying Technology of Messages
>>>> by Kannan Muthukkaruppan <http://www.facebook.com/Kannan>
>>>>
>>>>
>>>>
>>>> As a cassandra user I think the key sentence for this community is:
>>>> "We found Cassandra's eventual consistency model to be a difficult
>>>> pattern
>>>> to reconcile for our new Messages infrastructure."
>>>>
>>>> I think it would be useful to find out more about this statement from
>>>> Kannan
>>>> and the facebook team. Does anyone have any contacts in the Facebook
>>>> team?
>>>>
>>>> My goal here is to understand usage patterns and whether or not the
>>>> Cassandra community can learn from this decision; maybe even understand
>>>> whether the Cassandra roadmap should be influenced by this decision to
>>>> address a target user base. Of course we might also conclude that its
>>>> just
>>>> "not a Cassandra use-case"!
>>>>
>>>> Cheers,
>>>> Simon
>>>> --
>>>> Simon Reavely
>>>> simon.reavely@gmail.com
>>>
>>
>>
>
>
>
> On Sun, Nov 21, 2010 at 11:40 AM, David Boxenhorn <da...@lookin2.com> wrote:
>> Eventual consistency is not good enough for instant messaging.
>>
>> On Sun, Nov 21, 2010 at 6:32 PM, Simon Reavely <si...@gmail.com>
>> wrote:
>>>
>>> (Posting this to both user + dev lists)
>>>
>>> I was reviewing the blog post on the facebook engineering blog from nov
>>> 15th
>>> http://www.facebook.com/note.php?note_id=454991608919#
>>> <http://www.facebook.com/note.php?note_id=454991608919#>
>>> The Underlying Technology of Messages
>>> by Kannan Muthukkaruppan <http://www.facebook.com/Kannan>
>>>
>>>
>>>
>>> As a cassandra user I think the key sentence for this community is:
>>> "We found Cassandra's eventual consistency model to be a difficult pattern
>>> to reconcile for our new Messages infrastructure."
>>>
>>> I think it would be useful to find out more about this statement from
>>> Kannan
>>> and the facebook team. Does anyone have any contacts in the Facebook team?
>>>
>>> My goal here is to understand usage patterns and whether or not the
>>> Cassandra community can learn from this decision; maybe even understand
>>> whether the Cassandra roadmap should be influenced by this decision to
>>> address a target user base. Of course we might also conclude that its just
>>> "not a Cassandra use-case"!
>>>
>>> Cheers,
>>> Simon
>>> --
>>> Simon Reavely
>>> simon.reavely@gmail.com
>>
>>
>
> Jonathan Ellis pointed out a term that I like using better "Tunable
> consistency" . It seems that "eventual consistency" confuses everyone,
> that or it is an easy target of an anti Cassandra public relation
> campaign. If you want consistency use:
>
> WRITE.ALL + READ.ONE (hinted handoff off)
> WRITE.QUORUM + READ.QUORUM
> WRITE.ONE + READ.ALL
>
> Also I believe saying HBASE is consistent is not true. This can happen:
> Write to region server. -> Region Server acknowledges client-> write
> to WAL -> region server fails = write lost
>
> I wonder how facebook will reconcile that. :)
>
> Not trying to be nitpicky, at hadoop world in NYC I got to sit with
> lots of the hbase guys and we all had a great time talking about the
> mutual issues and happiness both of our communities share.
>
> We can not speak for Facebook, but likely chose HBase because they
> have several of hadoop core developers and have a large hadoop
> deployment. I would say the decision was probably based on several
> things. Current Cassandra release does not do on line schema updates.
> I am sure facebook does not want to restart 10,000 cassandra servers
> for a schema change. Current release does not have memtable tuning per
> column family. The upcoming Cassandra release has support for both of
> these things and many many more awesome things.
>
> Facebook is on the high end of how much data they have to manage, and
> how many servers they have. Most people do not share that use case. We
> can learn that facebook chose software that was good for them based on
> their use case and the experience they have in house. Something
> everyone should do.
Re: Facebook messaging and choice of HBase over Cassandra - what can we learn?
Posted by Aaron Morton <aa...@thelastpickle.com>.
I've been using "Tunable Consistency" when explaining Cassandra to people as well. I've also started pointing out the different Consistency models of the standard Transaction Isolation Levels of an ACID compliant DB, as well as "eventually consistent" async replication like MYSQL / Couch DB.
I've also been saying that consistency needs to be though of with regard to scope and time. For example one replica may be inconsistent to the rest of the replicas and you can still get a consistent read out. You can have inconsistent data in the system that may become consistent when read (via RR). But that the entire cluster (or perhaps it's better to say all replicas of the same data) will eventually store a consistent view of the truth.
But like you say, you *can* be consistent with regard to your reads and writes, when using the correct CL. The different combinations of read and write CL really interesting.
Cheers
Aaron
On 22 Nov, 2010,at 06:13 AM, Edward Capriolo <ed...@gmail.com> wrote:
On Sun, Nov 21, 2010 at 12:10 PM, André Fiedler
<fi...@googlemail.com> wrote:
> Facebook Messaging – HBase Comes of Age
>
> http://facility9.com/2010/11/18/facebook-messaging-hbase-comes-of-age
>
> 2010/11/21 David Boxenhorn <da...@lookin2.com>
>>
>> Eventual consistency is not good enough for instant messaging.
>>
>> On Sun, Nov 21, 2010 at 6:32 PM, Simon Reavely <si...@gmail.com>
>> wrote:
>>>
>>> (Posting this to both user + dev lists)
>>>
>>> I was reviewing the blog post on the facebook engineering blog from nov
>>> 15th
>>> http://www.facebook.com/note.php?note_id=454991608919#
>>> <http://www.facebook.com/note.php?note_id=454991608919#>
>>> The Underlying Technology of Messages
>>> by Kannan Muthukkaruppan <http://www.facebook.com/Kannan>
>>>
>>>
>>>
>>> As a cassandra user I think the key sentence for this community is:
>>> "We found Cassandra's eventual consistency model to be a difficult
>>> pattern
>>> to reconcile for our new Messages infrastructure."
>>>
>>> I think it would be useful to find out more about this statement from
>>> Kannan
>>> and the facebook team. Does anyone have any contacts in the Facebook
>>> team?
>>>
>>> My goal here is to understand usage patterns and whether or not the
>>> Cassandra community can learn from this decision; maybe even understand
>>> whether the Cassandra roadmap should be influenced by this decision to
>>> address a target user base. Of course we might also conclude that its
>>> just
>>> "not a Cassandra use-case"!
>>>
>>> Cheers,
>>> Simon
>>> --
>>> Simon Reavely
>>> simon.reavely@gmail.com
>>
>
>
On Sun, Nov 21, 2010 at 11:40 AM, David Boxenhorn <da...@lookin2.com> wrote:
> Eventual consistency is not good enough for instant messaging.
>
> On Sun, Nov 21, 2010 at 6:32 PM, Simon Reavely <si...@gmail.com>
> wrote:
>>
>> (Posting this to both user + dev lists)
>>
>> I was reviewing the blog post on the facebook engineering blog from nov
>> 15th
>> http://www.facebook.com/note.php?note_id=454991608919#
>> <http://www.facebook.com/note.php?note_id=454991608919#>
>> The Underlying Technology of Messages
>> by Kannan Muthukkaruppan <http://www.facebook.com/Kannan>
>>
>>
>>
>> As a cassandra user I think the key sentence for this community is:
>> "We found Cassandra's eventual consistency model to be a difficult pattern
>> to reconcile for our new Messages infrastructure."
>>
>> I think it would be useful to find out more about this statement from
>> Kannan
>> and the facebook team. Does anyone have any contacts in the Facebook team?
>>
>> My goal here is to understand usage patterns and whether or not the
>> Cassandra community can learn from this decision; maybe even understand
>> whether the Cassandra roadmap should be influenced by this decision to
>> address a target user base. Of course we might also conclude that its just
>> "not a Cassandra use-case"!
>>
>> Cheers,
>> Simon
>> --
>> Simon Reavely
>> simon.reavely@gmail.com
>
>
Jonathan Ellis pointed out a term that I like using better "Tunable
consistency" . It seems that "eventual consistency" confuses everyone,
that or it is an easy target of an anti Cassandra public relation
campaign. If you want consistency use:
WRITE.ALL + READ.ONE (hinted handoff off)
WRITE.QUORUM + READ.QUORUM
WRITE.ONE + READ.ALL
Also I believe saying HBASE is consistent is not true. This can happen:
Write to region server. -> Region Server acknowledges client-> write
to WAL -> region server fails = write lost
I wonder how facebook will reconcile that. :)
Not trying to be nitpicky, at hadoop world in NYC I got to sit with
lots of the hbase guys and we all had a great time talking about the
mutual issues and happiness both of our communities share.
We can not speak for Facebook, but likely chose HBase because they
have several of hadoop core developers and have a large hadoop
deployment. I would say the decision was probably based on several
things. Current Cassandra release does not do on line schema updates.
I am sure facebook does not want to restart 10,000 cassandra servers
for a schema change. Current release does not have memtable tuning per
column family. The upcoming Cassandra release has support for both of
these things and many many more awesome things.
Facebook is on the high end of how much data they have to manage, and
how many servers they have. Most people do not share that use case. We
can learn that facebook chose software that was good for them based on
their use case and the experience they have in house. Something
everyone should do.
Re: Facebook messaging and choice of HBase over Cassandra - what can
we learn?
Posted by Edward Capriolo <ed...@gmail.com>.
On Sun, Nov 21, 2010 at 12:10 PM, André Fiedler
<fi...@googlemail.com> wrote:
> Facebook Messaging – HBase Comes of Age
>
> http://facility9.com/2010/11/18/facebook-messaging-hbase-comes-of-age
>
> 2010/11/21 David Boxenhorn <da...@lookin2.com>
>>
>> Eventual consistency is not good enough for instant messaging.
>>
>> On Sun, Nov 21, 2010 at 6:32 PM, Simon Reavely <si...@gmail.com>
>> wrote:
>>>
>>> (Posting this to both user + dev lists)
>>>
>>> I was reviewing the blog post on the facebook engineering blog from nov
>>> 15th
>>> http://www.facebook.com/note.php?note_id=454991608919#
>>> <http://www.facebook.com/note.php?note_id=454991608919#>
>>> The Underlying Technology of Messages
>>> by Kannan Muthukkaruppan <http://www.facebook.com/Kannan>
>>>
>>>
>>>
>>> As a cassandra user I think the key sentence for this community is:
>>> "We found Cassandra's eventual consistency model to be a difficult
>>> pattern
>>> to reconcile for our new Messages infrastructure."
>>>
>>> I think it would be useful to find out more about this statement from
>>> Kannan
>>> and the facebook team. Does anyone have any contacts in the Facebook
>>> team?
>>>
>>> My goal here is to understand usage patterns and whether or not the
>>> Cassandra community can learn from this decision; maybe even understand
>>> whether the Cassandra roadmap should be influenced by this decision to
>>> address a target user base. Of course we might also conclude that its
>>> just
>>> "not a Cassandra use-case"!
>>>
>>> Cheers,
>>> Simon
>>> --
>>> Simon Reavely
>>> simon.reavely@gmail.com
>>
>
>
On Sun, Nov 21, 2010 at 11:40 AM, David Boxenhorn <da...@lookin2.com> wrote:
> Eventual consistency is not good enough for instant messaging.
>
> On Sun, Nov 21, 2010 at 6:32 PM, Simon Reavely <si...@gmail.com>
> wrote:
>>
>> (Posting this to both user + dev lists)
>>
>> I was reviewing the blog post on the facebook engineering blog from nov
>> 15th
>> http://www.facebook.com/note.php?note_id=454991608919#
>> <http://www.facebook.com/note.php?note_id=454991608919#>
>> The Underlying Technology of Messages
>> by Kannan Muthukkaruppan <http://www.facebook.com/Kannan>
>>
>>
>>
>> As a cassandra user I think the key sentence for this community is:
>> "We found Cassandra's eventual consistency model to be a difficult pattern
>> to reconcile for our new Messages infrastructure."
>>
>> I think it would be useful to find out more about this statement from
>> Kannan
>> and the facebook team. Does anyone have any contacts in the Facebook team?
>>
>> My goal here is to understand usage patterns and whether or not the
>> Cassandra community can learn from this decision; maybe even understand
>> whether the Cassandra roadmap should be influenced by this decision to
>> address a target user base. Of course we might also conclude that its just
>> "not a Cassandra use-case"!
>>
>> Cheers,
>> Simon
>> --
>> Simon Reavely
>> simon.reavely@gmail.com
>
>
Jonathan Ellis pointed out a term that I like using better "Tunable
consistency" . It seems that "eventual consistency" confuses everyone,
that or it is an easy target of an anti Cassandra public relation
campaign. If you want consistency use:
WRITE.ALL + READ.ONE (hinted handoff off)
WRITE.QUORUM + READ.QUORUM
WRITE.ONE + READ.ALL
Also I believe saying HBASE is consistent is not true. This can happen:
Write to region server. -> Region Server acknowledges client-> write
to WAL -> region server fails = write lost
I wonder how facebook will reconcile that. :)
Not trying to be nitpicky, at hadoop world in NYC I got to sit with
lots of the hbase guys and we all had a great time talking about the
mutual issues and happiness both of our communities share.
We can not speak for Facebook, but likely chose HBase because they
have several of hadoop core developers and have a large hadoop
deployment. I would say the decision was probably based on several
things. Current Cassandra release does not do on line schema updates.
I am sure facebook does not want to restart 10,000 cassandra servers
for a schema change. Current release does not have memtable tuning per
column family. The upcoming Cassandra release has support for both of
these things and many many more awesome things.
Facebook is on the high end of how much data they have to manage, and
how many servers they have. Most people do not share that use case. We
can learn that facebook chose software that was good for them based on
their use case and the experience they have in house. Something
everyone should do.
Re: Facebook messaging and choice of HBase over Cassandra - what can
we learn?
Posted by André Fiedler <fi...@googlemail.com>.
Facebook Messaging – HBase Comes of
Age<http://facility9.com/2010/11/18/facebook-messaging-hbase-comes-of-age>
http://facility9.com/2010/11/18/facebook-messaging-hbase-comes-of-age
2010/11/21 David Boxenhorn <da...@lookin2.com>
> Eventual consistency is not good enough for instant messaging.
>
> On Sun, Nov 21, 2010 at 6:32 PM, Simon Reavely <si...@gmail.com>wrote:
>
>> (Posting this to both user + dev lists)
>>
>> I was reviewing the blog post on the facebook engineering blog from nov
>> 15th
>> http://www.facebook.com/note.php?note_id=454991608919#
>> <http://www.facebook.com/note.php?note_id=454991608919#>
>> The Underlying Technology of Messages
>> by Kannan Muthukkaruppan <http://www.facebook.com/Kannan>
>>
>>
>>
>>
>> As a cassandra user I think the key sentence for this community is:
>> "We found Cassandra's eventual consistency model to be a difficult pattern
>> to reconcile for our new Messages infrastructure."
>>
>> I think it would be useful to find out more about this statement from
>> Kannan
>> and the facebook team. Does anyone have any contacts in the Facebook team?
>>
>> My goal here is to understand usage patterns and whether or not the
>> Cassandra community can learn from this decision; maybe even understand
>> whether the Cassandra roadmap should be influenced by this decision to
>> address a target user base. Of course we might also conclude that its just
>> "not a Cassandra use-case"!
>>
>> Cheers,
>> Simon
>> --
>> Simon Reavely
>> simon.reavely@gmail.com
>>
>
>
Re: Facebook messaging and choice of HBase over Cassandra - what can
we learn?
Posted by David Boxenhorn <da...@lookin2.com>.
Eventual consistency is not good enough for instant messaging.
On Sun, Nov 21, 2010 at 6:32 PM, Simon Reavely <si...@gmail.com>wrote:
> (Posting this to both user + dev lists)
>
> I was reviewing the blog post on the facebook engineering blog from nov
> 15th
> http://www.facebook.com/note.php?note_id=454991608919#
> <http://www.facebook.com/note.php?note_id=454991608919#>
> The Underlying Technology of Messages
> by Kannan Muthukkaruppan <http://www.facebook.com/Kannan>
>
>
>
> As a cassandra user I think the key sentence for this community is:
> "We found Cassandra's eventual consistency model to be a difficult pattern
> to reconcile for our new Messages infrastructure."
>
> I think it would be useful to find out more about this statement from
> Kannan
> and the facebook team. Does anyone have any contacts in the Facebook team?
>
> My goal here is to understand usage patterns and whether or not the
> Cassandra community can learn from this decision; maybe even understand
> whether the Cassandra roadmap should be influenced by this decision to
> address a target user base. Of course we might also conclude that its just
> "not a Cassandra use-case"!
>
> Cheers,
> Simon
> --
> Simon Reavely
> simon.reavely@gmail.com
>
Re: Facebook messaging and choice of HBase over Cassandra - what can
we learn?
Posted by David Boxenhorn <da...@lookin2.com>.
Eventual consistency is not good enough for instant messaging.
On Sun, Nov 21, 2010 at 6:32 PM, Simon Reavely <si...@gmail.com>wrote:
> (Posting this to both user + dev lists)
>
> I was reviewing the blog post on the facebook engineering blog from nov
> 15th
> http://www.facebook.com/note.php?note_id=454991608919#
> <http://www.facebook.com/note.php?note_id=454991608919#>
> The Underlying Technology of Messages
> by Kannan Muthukkaruppan <http://www.facebook.com/Kannan>
>
>
>
> As a cassandra user I think the key sentence for this community is:
> "We found Cassandra's eventual consistency model to be a difficult pattern
> to reconcile for our new Messages infrastructure."
>
> I think it would be useful to find out more about this statement from
> Kannan
> and the facebook team. Does anyone have any contacts in the Facebook team?
>
> My goal here is to understand usage patterns and whether or not the
> Cassandra community can learn from this decision; maybe even understand
> whether the Cassandra roadmap should be influenced by this decision to
> address a target user base. Of course we might also conclude that its just
> "not a Cassandra use-case"!
>
> Cheers,
> Simon
> --
> Simon Reavely
> simon.reavely@gmail.com
>