You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Mimi Aluminium <mi...@gmail.com> on 2011/03/01 20:47:56 UTC
how large can a cluster over the WAN be?
Hi,
Are there clusters of 100 nodes? more? Please can you refer me to such
installations/ systems?
Can you comment on over-the-WAN clusters in this size or less? and can you
point on system with nodes in different DCs connected by WAN ( could be
dedicated or internet) ?
Thanks a lot,
Miriam
Miriam Allalouf
Miriam Allalouf, PhD
----------------------------------------
Storage & Network Research
IBM, Haifa Research Labs
Re: how large can a cluster over the WAN be?
Posted by John Lewis <le...@gmail.com>.
Thanks for the reply, I realize my question was rather nebulous as I consider this proposed deployment to be rather nebulous as well. Any bit of information and a direction on which sections of documentation are relevant helps this challenge become less nebulous over time. I will do some reading on the topics you have provided.
Thanks again
Lewis
On Mar 7, 2011, at 11:52 PM, Robert Coli wrote:
> On Mon, Mar 7, 2011 at 11:32 AM, John Lewis <le...@gmail.com> wrote:
>> When you say decent latency and throughput what numbers do you consider decent? I know throughput would be highly dependent on the quantity of kb shoved through the pipe so I would expect throughput needs would be highly dependent on the data actually in cassandra.
>
> As you say, throughput needed is dependent on Cassandra payload size,
> but also (in 0.7) read repair percentage. Cassandra is a large
> consumer of network traffic relative to the amount of data serviced to
> clients due to background repair processes like read repair and manual
> AES repair. There are obviously scenarios where you might saturate the
> WAN link given large enough nodes or numbers of nodes per datacenter..
>
> When I am talking about latency, my experience is with WAN latency
> under 100ms and without DynamicEndpointSnitch. I suspect that within
> an order of magnitude of that latency, with or without DES, is likely
> to be fine for many use cases. There are a few tunables which might
> be appropriate to increase when operating in more than two datacenters
> with greater possible latency between any two as well as replication
> strategies and consistency levels which offer certain latency
> behavior. As always, simulating your actual workload is likely to give
> you the most relevant information as to the impact of inter-cassandra
> latency on your application. :)
>
> =Rob
Re: how large can a cluster over the WAN be?
Posted by Robert Coli <rc...@digg.com>.
On Mon, Mar 7, 2011 at 11:32 AM, John Lewis <le...@gmail.com> wrote:
> When you say decent latency and throughput what numbers do you consider decent? I know throughput would be highly dependent on the quantity of kb shoved through the pipe so I would expect throughput needs would be highly dependent on the data actually in cassandra.
As you say, throughput needed is dependent on Cassandra payload size,
but also (in 0.7) read repair percentage. Cassandra is a large
consumer of network traffic relative to the amount of data serviced to
clients due to background repair processes like read repair and manual
AES repair. There are obviously scenarios where you might saturate the
WAN link given large enough nodes or numbers of nodes per datacenter..
When I am talking about latency, my experience is with WAN latency
under 100ms and without DynamicEndpointSnitch. I suspect that within
an order of magnitude of that latency, with or without DES, is likely
to be fine for many use cases. There are a few tunables which might
be appropriate to increase when operating in more than two datacenters
with greater possible latency between any two as well as replication
strategies and consistency levels which offer certain latency
behavior. As always, simulating your actual workload is likely to give
you the most relevant information as to the impact of inter-cassandra
latency on your application. :)
=Rob
Re: how large can a cluster over the WAN be?
Posted by John Lewis <le...@gmail.com>.
When you say decent latency and throughput what numbers do you consider decent? I know throughput would be highly dependent on the quantity of kb shoved through the pipe so I would expect throughput needs would be highly dependent on the data actually in cassandra.
Thanks for the info, I am getting ready for a multisite deployment and any bit of information will help us qualify or disqualify network vendors with regards to what we anticipate our needs to be.
Lewis
On Mar 7, 2011, at 10:25 AM, Robert Coli wrote:
> On Sun, Mar 6, 2011 at 1:39 PM, Mimi Aluminium <mi...@gmail.com> wrote:
>> Are you familiar with Cassandra cluster that is installed in datacenters
>> that are spread across the WAN? can you comment on the perfromance of such
>> installation?
>> What is the largest size of of such a cluster you are aware of?
>
> Digg operated a 40 node cassandra 0.6.1 to 0.6.6+1072 cluster in two
> datacenters, one on each coast of the US. It worked fine, with the
> usual caveats that come with that sort of network latency. We used a
> custom snitch and rackaware, but the implementation in 0.6.x was
> insufficiently robust and ended up disabled in favor of simple snitch
> and rack unaware with the nodes simply alternating data centers on the
> ring. As we only read from one half of the cluster, this meant that we
> often only had one local replica of our data. We recently moved this
> cluster from two physical DCs to one. Other than the network trickery
> involved in keeping IP addresses the same, the only negative aspect of
> the WAN in this case was the bottleneck while copying the data.
>
> As long as you have decent latency and throughput across the WAN link,
> cassandra should be fine. This should be especially true in 0.7 and
> with the DynamicEndpointSnitch enabled. I have clearly used a number
> of weasel words in this summary, you should of course test for your
> case, especially if that case involves more than two datacenters.
>
> =Rob
Re: how large can a cluster over the WAN be?
Posted by Robert Coli <rc...@digg.com>.
On Sun, Mar 6, 2011 at 1:39 PM, Mimi Aluminium <mi...@gmail.com> wrote:
> Are you familiar with Cassandra cluster that is installed in datacenters
> that are spread across the WAN? can you comment on the perfromance of such
> installation?
> What is the largest size of of such a cluster you are aware of?
Digg operated a 40 node cassandra 0.6.1 to 0.6.6+1072 cluster in two
datacenters, one on each coast of the US. It worked fine, with the
usual caveats that come with that sort of network latency. We used a
custom snitch and rackaware, but the implementation in 0.6.x was
insufficiently robust and ended up disabled in favor of simple snitch
and rack unaware with the nodes simply alternating data centers on the
ring. As we only read from one half of the cluster, this meant that we
often only had one local replica of our data. We recently moved this
cluster from two physical DCs to one. Other than the network trickery
involved in keeping IP addresses the same, the only negative aspect of
the WAN in this case was the bottleneck while copying the data.
As long as you have decent latency and throughput across the WAN link,
cassandra should be fine. This should be especially true in 0.7 and
with the DynamicEndpointSnitch enabled. I have clearly used a number
of weasel words in this summary, you should of course test for your
case, especially if that case involves more than two datacenters.
=Rob
Re: Designing a decent data model for an online music
shop...confused/stuck on decisions
Posted by Sasha Dolgy <sd...@gmail.com>.
phpcassa: https://github.com/thobbs/phpcassa is maintained and works well
...
-sd
On Mon, Mar 7, 2011 at 4:22 PM, Courtney <e-...@hotmail.com> wrote:
> Thanks for the response, I haven't checked on the status of phpcassa in a
> while but does it now work with 0.7?
> That was one of the main reasons I switched to pandra, it seemed more up to
> date
>
> *From:* Tyler Hobbs <ty...@datastax.com>
> *Sent:* Monday, March 07, 2011 2:40 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Designing a decent data model for an online music
> shop...confused/stuck on decisions
>
> Regarding PHP performance with Cassandra, THRIFT-638<https://issues.apache.org/jira/browse/THRIFT-638>was recently resolved and it shows some big performance improvements. I'll
> be upgrading the Thrift package that ships with phpcassa soon to include
> this fix, so you may want to compare performance numbers before and after.
>
> On Sun, Mar 6, 2011 at 8:03 PM, Courtney <e-...@hotmail.com> wrote:
>
>> We're in a bit of a predicament, we have an e-music store currently
>> built in PHP using codeigniter/mysql...
>> The current system has 100+K users and a decent song collection. Over the
>> last few months I've been playing with
>> Cassandra... needless to say I'm impressed but I have a few questions.
>> Firstly, I want to avoid re-writing the entire site if possible so my
>> instincts have made me inclined to replace the database layer
>> in code igniter... is this something anyone would recommend and are there
>> any gotchas in doing that?
>>
>> I can't say I've been terribly happy with PHP accessing cassandra, when
>> sample data of the same size was put into mysql and in cassandra (of the
>> same size/type)
>> The pages with php connecting to Cassandra took longer to load, (30K
>> records in table).
>> I've thought maybe it was my setup that needed tweaking and I've played
>> with as many a options as I could but the best I've gotten is matching query
>> time.
>> Query speed test was simply getting time stamps right before and after
>> query call returned...
>>
>> Is this something anyone else has seen, any comments suggestions? I've
>> tried using thrift, phpcassa and pandra with pretty similar numbers.
>>
>> My other thought turned to maybe it was the way I designed my CFs, at
>> first I used super columns to model user account CF based on a post I read
>> by Arin (WTF is a super column) but I later changed to using normal CFs.
>>
>> I'm trying to make this work but I get the feeling my approach is
>> somewhat...I don't mis-guided.
>>
>> Here's a break down of the current model.
>> CF:Users{
>> uid
>> fname
>> lname
>> username
>> password
>> street
>> ....
>> }
>> Some additional columns in place for a user but keeping it simple...
>> CF:Library{
>> uid
>> songid
>> ...
>> other info about user library
>> }
>>
>> CF:Songs{
>> songid
>> title
>> artistid
>> }
>>
>> This all is still very relational like (considering I go on to have a CF
>> for playlist and artists) and I'm not sure if this is a good design for the
>> data but... when I looked into
>> combining some of the info and removing some CFs I run into the issue of
>> replicating data all over the place. If for example I stored the artist name
>> in the library for each record
>> then each then the artist would be replicated for every song they have for
>> every user who has that song in their library....
>>
>> Where do you sort of draw the line on deciding how much is okay to be
>> replicated?
>>
>> As much as I am not liking the idea of building the application from
>> scratch, I'm considering the possibility of building from scratch in
>> Java/JSP just to get the benefit of using
>> the hector client. (Efforts from the guys doing the PHP libs is much
>> appreciated but PHP doesn't seem to go too well with Cas.)
>>
>> In the process of making decisions because the upgrade/rebuild needs to
>> have a fairly steady working version for October and I don't want to go
>> wrong before even starting.
>>
>> Recommendations. Suggestions, advice are all welcomed (Any experience with
>> PHP and Cas. is also welcomed since all my fav. libs. are in PHP I'm
>> reluctant to turn away)
>>
>
>
>
> --
> Tyler Hobbs
> Software Engineer, DataStax <http://datastax.com/>
> Maintainer of the pycassa <http://github.com/pycassa/pycassa> Cassandra
> Python client library
>
>
--
Sasha Dolgy
sasha.dolgy@gmail.com
Re: Designing a decent data model for an online music shop...confused/stuck on decisions
Posted by Courtney <e-...@hotmail.com>.
Thanks for the response, I haven't checked on the status of phpcassa in a while but does it now work with 0.7?
That was one of the main reasons I switched to pandra, it seemed more up to date
From: Tyler Hobbs
Sent: Monday, March 07, 2011 2:40 AM
To: user@cassandra.apache.org
Subject: Re: Designing a decent data model for an online music shop...confused/stuck on decisions
Regarding PHP performance with Cassandra, THRIFT-638 was recently resolved and it shows some big performance improvements. I'll be upgrading the Thrift package that ships with phpcassa soon to include this fix, so you may want to compare performance numbers before and after.
On Sun, Mar 6, 2011 at 8:03 PM, Courtney <e-...@hotmail.com> wrote:
We're in a bit of a predicament, we have an e-music store currently built in PHP using codeigniter/mysql...
The current system has 100+K users and a decent song collection. Over the last few months I've been playing with
Cassandra... needless to say I'm impressed but I have a few questions.
Firstly, I want to avoid re-writing the entire site if possible so my instincts have made me inclined to replace the database layer
in code igniter... is this something anyone would recommend and are there any gotchas in doing that?
I can't say I've been terribly happy with PHP accessing cassandra, when sample data of the same size was put into mysql and in cassandra (of the same size/type)
The pages with php connecting to Cassandra took longer to load, (30K records in table).
I've thought maybe it was my setup that needed tweaking and I've played with as many a options as I could but the best I've gotten is matching query time.
Query speed test was simply getting time stamps right before and after query call returned...
Is this something anyone else has seen, any comments suggestions? I've tried using thrift, phpcassa and pandra with pretty similar numbers.
My other thought turned to maybe it was the way I designed my CFs, at first I used super columns to model user account CF based on a post I read
by Arin (WTF is a super column) but I later changed to using normal CFs.
I'm trying to make this work but I get the feeling my approach is somewhat...I don't mis-guided.
Here's a break down of the current model.
CF:Users{
uid
fname
lname
username
password
street
....
}
Some additional columns in place for a user but keeping it simple...
CF:Library{
uid
songid
...
other info about user library
}
CF:Songs{
songid
title
artistid
}
This all is still very relational like (considering I go on to have a CF for playlist and artists) and I'm not sure if this is a good design for the data but... when I looked into
combining some of the info and removing some CFs I run into the issue of replicating data all over the place. If for example I stored the artist name in the library for each record
then each then the artist would be replicated for every song they have for every user who has that song in their library....
Where do you sort of draw the line on deciding how much is okay to be replicated?
As much as I am not liking the idea of building the application from scratch, I'm considering the possibility of building from scratch in Java/JSP just to get the benefit of using
the hector client. (Efforts from the guys doing the PHP libs is much appreciated but PHP doesn't seem to go too well with Cas.)
In the process of making decisions because the upgrade/rebuild needs to have a fairly steady working version for October and I don't want to go wrong before even starting.
Recommendations. Suggestions, advice are all welcomed (Any experience with PHP and Cas. is also welcomed since all my fav. libs. are in PHP I'm reluctant to turn away)
--
Tyler Hobbs
Software Engineer, DataStax
Maintainer of the pycassa Cassandra Python client library
Re: Designing a decent data model for an online music
shop...confused/stuck on decisions
Posted by Tyler Hobbs <ty...@datastax.com>.
Regarding PHP performance with Cassandra,
THRIFT-638<https://issues.apache.org/jira/browse/THRIFT-638>was
recently resolved and it shows some big performance improvements.
I'll
be upgrading the Thrift package that ships with phpcassa soon to include
this fix, so you may want to compare performance numbers before and after.
On Sun, Mar 6, 2011 at 8:03 PM, Courtney <e-...@hotmail.com> wrote:
> We're in a bit of a predicament, we have an e-music store currently built
> in PHP using codeigniter/mysql...
> The current system has 100+K users and a decent song collection. Over the
> last few months I've been playing with
> Cassandra... needless to say I'm impressed but I have a few questions.
> Firstly, I want to avoid re-writing the entire site if possible so my
> instincts have made me inclined to replace the database layer
> in code igniter... is this something anyone would recommend and are there
> any gotchas in doing that?
>
> I can't say I've been terribly happy with PHP accessing cassandra, when
> sample data of the same size was put into mysql and in cassandra (of the
> same size/type)
> The pages with php connecting to Cassandra took longer to load, (30K
> records in table).
> I've thought maybe it was my setup that needed tweaking and I've played
> with as many a options as I could but the best I've gotten is matching query
> time.
> Query speed test was simply getting time stamps right before and after
> query call returned...
>
> Is this something anyone else has seen, any comments suggestions? I've
> tried using thrift, phpcassa and pandra with pretty similar numbers.
>
> My other thought turned to maybe it was the way I designed my CFs, at first
> I used super columns to model user account CF based on a post I read
> by Arin (WTF is a super column) but I later changed to using normal CFs.
>
> I'm trying to make this work but I get the feeling my approach is
> somewhat...I don't mis-guided.
>
> Here's a break down of the current model.
> CF:Users{
> uid
> fname
> lname
> username
> password
> street
> ....
> }
> Some additional columns in place for a user but keeping it simple...
> CF:Library{
> uid
> songid
> ...
> other info about user library
> }
>
> CF:Songs{
> songid
> title
> artistid
> }
>
> This all is still very relational like (considering I go on to have a CF
> for playlist and artists) and I'm not sure if this is a good design for the
> data but... when I looked into
> combining some of the info and removing some CFs I run into the issue of
> replicating data all over the place. If for example I stored the artist name
> in the library for each record
> then each then the artist would be replicated for every song they have for
> every user who has that song in their library....
>
> Where do you sort of draw the line on deciding how much is okay to be
> replicated?
>
> As much as I am not liking the idea of building the application from
> scratch, I'm considering the possibility of building from scratch in
> Java/JSP just to get the benefit of using
> the hector client. (Efforts from the guys doing the PHP libs is much
> appreciated but PHP doesn't seem to go too well with Cas.)
>
> In the process of making decisions because the upgrade/rebuild needs to
> have a fairly steady working version for October and I don't want to go
> wrong before even starting.
>
> Recommendations. Suggestions, advice are all welcomed (Any experience with
> PHP and Cas. is also welcomed since all my fav. libs. are in PHP I'm
> reluctant to turn away)
>
--
Tyler Hobbs
Software Engineer, DataStax <http://datastax.com/>
Maintainer of the pycassa <http://github.com/pycassa/pycassa> Cassandra
Python client library
Designing a decent data model for an online music shop...confused/stuck on decisions
Posted by Courtney <e-...@hotmail.com>.
We're in a bit of a predicament, we have an e-music store currently built in PHP using codeigniter/mysql...
The current system has 100+K users and a decent song collection. Over the last few months I've been playing with
Cassandra... needless to say I'm impressed but I have a few questions.
Firstly, I want to avoid re-writing the entire site if possible so my instincts have made me inclined to replace the database layer
in code igniter... is this something anyone would recommend and are there any gotchas in doing that?
I can't say I've been terribly happy with PHP accessing cassandra, when sample data of the same size was put into mysql and in cassandra (of the same size/type)
The pages with php connecting to Cassandra took longer to load, (30K records in table).
I've thought maybe it was my setup that needed tweaking and I've played with as many a options as I could but the best I've gotten is matching query time.
Query speed test was simply getting time stamps right before and after query call returned...
Is this something anyone else has seen, any comments suggestions? I've tried using thrift, phpcassa and pandra with pretty similar numbers.
My other thought turned to maybe it was the way I designed my CFs, at first I used super columns to model user account CF based on a post I read
by Arin (WTF is a super column) but I later changed to using normal CFs.
I'm trying to make this work but I get the feeling my approach is somewhat...I don't mis-guided.
Here's a break down of the current model.
CF:Users{
uid
fname
lname
username
password
street
....
}
Some additional columns in place for a user but keeping it simple...
CF:Library{
uid
songid
...
other info about user library
}
CF:Songs{
songid
title
artistid
}
This all is still very relational like (considering I go on to have a CF for playlist and artists) and I'm not sure if this is a good design for the data but... when I looked into
combining some of the info and removing some CFs I run into the issue of replicating data all over the place. If for example I stored the artist name in the library for each record
then each then the artist would be replicated for every song they have for every user who has that song in their library....
Where do you sort of draw the line on deciding how much is okay to be replicated?
As much as I am not liking the idea of building the application from scratch, I'm considering the possibility of building from scratch in Java/JSP just to get the benefit of using
the hector client. (Efforts from the guys doing the PHP libs is much appreciated but PHP doesn't seem to go too well with Cas.)
In the process of making decisions because the upgrade/rebuild needs to have a fairly steady working version for October and I don't want to go wrong before even starting.
Recommendations. Suggestions, advice are all welcomed (Any experience with PHP and Cas. is also welcomed since all my fav. libs. are in PHP I'm reluctant to turn away)
Re: how large can a cluster over the WAN be?
Posted by Mimi Aluminium <mi...@gmail.com>.
Hi,
Please, can you help with the following? it will lead us in some design
decisions.
Are you familiar with Cassandra cluster that is installed in datacenters
that are spread across the WAN? can you comment on the perfromance of such
installation?
What is the largest size of of such a cluster you are aware of?
Thanks a lot,
Miriam
On Tue, Mar 1, 2011 at 9:47 PM, Mimi Aluminium <mi...@gmail.com>wrote:
> Hi,
> Are there clusters of 100 nodes? more? Please can you refer me to such
> installations/ systems?
> Can you comment on over-the-WAN clusters in this size or less? and can you
> point on system with nodes in different DCs connected by WAN ( could be
> dedicated or internet) ?
> Thanks a lot,
> Miriam
>
>
>
>
>