You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Suhail Doshi <su...@mixpanel.com> on 2010/01/29 04:23:47 UTC
Understand how to provision nodes and use cassandra in the production
We've started to use Cassandra in production and just have one node right
now. Here's one of our ColumnFamilys:
16G Jan 28 22:28 SomeIndex-5467-Index.db
196M Jan 28 22:32 SomeIndex-5487-Index.db
The first bottle neck you encounter is reads--writes are extremely
fast even with one node.
My question is, is the size of the *-Index.db files the amount of RAM
you need available for Cassandra to do reads fast?
What are some configuration options you would need to tweak besides
the JVM's max memory size being larger. Is there any default
configurations commonly missed?
Next, if you provision more nodes will Cassandra distribute the data
in memory so I don't need a single 16 GB node? Is there anything I
need to build in my application logic to make this work correctly.
Ideally, if I had a 16 GB index, I'd want it spread across 4 4GB
nodes. Can any client connect to any one node request info and it will
get the info back from a node that has that part of the index in
memory?
What's the best way to do efficient reads?
Suhail
Re: Understand how to provision nodes and use cassandra in the
production
Posted by Suhail Doshi <di...@gmail.com>.
Additionally:
Address Status Load Range
Ring
85079825064071324593650466313420553448
<seed_ip>Up 58.08 GB 25804699734015282125022172898213238764
|<--|
<non_seed>Up 19.71 GB 85079825064071324593650466313420553448
|-->|
On Sat, Jan 30, 2010 at 3:32 AM, Suhail Doshi <di...@gmail.com>wrote:
> I should note the new node has been bootstrapped and the data has been
> distributed which further perplexes me.
>
> The index file I am reading off is about 16G
>
>
> On Sat, Jan 30, 2010 at 3:23 AM, Suhail Doshi <di...@gmail.com>wrote:
>
>> An issue I've been seeing is it's really hard to scale Cassandra with
>> reads. I've run top, vmstat, iostat. vmstat shows no swapping but iostat
>> shows heavy saturation of %util and await times over 90ms with max rMB/s of
>> 7-8.
>>
>> I have over 7G of memory dedicated across two nodes. I am wondering what
>> the issue might be and how to solve this? I felt like 7 G would be enough.
>>
>> Suhail
>>
>>
>> On Thu, Jan 28, 2010 at 7:32 PM, Ray Slakinski <ra...@mahalo.com> wrote:
>>
>>> Cassandra auto shards, so you just need to point at your cluster and
>>> cassandra does the rest. You should read up on different partitioners though
>>> before you go live in production, because its not too easy to switch once
>>> you make that decision.
>>>
>>> http://wiki.apache.org/cassandra/StorageConfiguration#Partitioner
>>>
>>> Ray Slakinski
>>> On 2010-01-28, at 7:29 PM, Suhail Doshi wrote:
>>>
>>> > Another piece I am interested in is how cassandra distributes the data
>>> > automatically. In MySQL you need to shard and you'd pick the shard to
>>> > request info from--how does that translate in cassandra?
>>> >
>>> > On Thu, Jan 28, 2010 at 7:23 PM, Suhail Doshi <su...@mixpanel.com>
>>> wrote:
>>> >
>>> >> We've started to use Cassandra in production and just have one node
>>> right
>>> >> now. Here's one of our ColumnFamilys:
>>> >>
>>> >> 16G Jan 28 22:28 SomeIndex-5467-Index.db
>>> >> 196M Jan 28 22:32 SomeIndex-5487-Index.db
>>> >>
>>> >> The first bottle neck you encounter is reads--writes are extremely
>>> fast even with one node.
>>> >>
>>> >> My question is, is the size of the *-Index.db files the amount of RAM
>>> you need available for Cassandra to do reads fast?
>>> >>
>>> >> What are some configuration options you would need to tweak besides
>>> the JVM's max memory size being larger. Is there any default configurations
>>> commonly missed?
>>> >>
>>> >> Next, if you provision more nodes will Cassandra distribute the data
>>> in memory so I don't need a single 16 GB node? Is there anything I need to
>>> build in my application logic to make this work correctly. Ideally, if I had
>>> a 16 GB index, I'd want it spread across 4 4GB nodes. Can any client connect
>>> to any one node request info and it will get the info back from a node that
>>> has that part of the index in memory?
>>> >>
>>> >> What's the best way to do efficient reads?
>>> >>
>>> >> Suhail
>>> >>
>>> >>
>>>
>>>
>>
>>
>> --
>> http://mixpanel.com
>> Blog: http://blog.mixpanel.com
>>
>
>
>
> --
> http://mixpanel.com
> Blog: http://blog.mixpanel.com
>
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
Re: Understand how to provision nodes and use cassandra in the
production
Posted by Suhail Doshi <di...@gmail.com>.
I should note the new node has been bootstrapped and the data has been
distributed which further perplexes me.
The index file I am reading off is about 16G
On Sat, Jan 30, 2010 at 3:23 AM, Suhail Doshi <di...@gmail.com>wrote:
> An issue I've been seeing is it's really hard to scale Cassandra with
> reads. I've run top, vmstat, iostat. vmstat shows no swapping but iostat
> shows heavy saturation of %util and await times over 90ms with max rMB/s of
> 7-8.
>
> I have over 7G of memory dedicated across two nodes. I am wondering what
> the issue might be and how to solve this? I felt like 7 G would be enough.
>
> Suhail
>
>
> On Thu, Jan 28, 2010 at 7:32 PM, Ray Slakinski <ra...@mahalo.com> wrote:
>
>> Cassandra auto shards, so you just need to point at your cluster and
>> cassandra does the rest. You should read up on different partitioners though
>> before you go live in production, because its not too easy to switch once
>> you make that decision.
>>
>> http://wiki.apache.org/cassandra/StorageConfiguration#Partitioner
>>
>> Ray Slakinski
>> On 2010-01-28, at 7:29 PM, Suhail Doshi wrote:
>>
>> > Another piece I am interested in is how cassandra distributes the data
>> > automatically. In MySQL you need to shard and you'd pick the shard to
>> > request info from--how does that translate in cassandra?
>> >
>> > On Thu, Jan 28, 2010 at 7:23 PM, Suhail Doshi <su...@mixpanel.com>
>> wrote:
>> >
>> >> We've started to use Cassandra in production and just have one node
>> right
>> >> now. Here's one of our ColumnFamilys:
>> >>
>> >> 16G Jan 28 22:28 SomeIndex-5467-Index.db
>> >> 196M Jan 28 22:32 SomeIndex-5487-Index.db
>> >>
>> >> The first bottle neck you encounter is reads--writes are extremely fast
>> even with one node.
>> >>
>> >> My question is, is the size of the *-Index.db files the amount of RAM
>> you need available for Cassandra to do reads fast?
>> >>
>> >> What are some configuration options you would need to tweak besides the
>> JVM's max memory size being larger. Is there any default configurations
>> commonly missed?
>> >>
>> >> Next, if you provision more nodes will Cassandra distribute the data in
>> memory so I don't need a single 16 GB node? Is there anything I need to
>> build in my application logic to make this work correctly. Ideally, if I had
>> a 16 GB index, I'd want it spread across 4 4GB nodes. Can any client connect
>> to any one node request info and it will get the info back from a node that
>> has that part of the index in memory?
>> >>
>> >> What's the best way to do efficient reads?
>> >>
>> >> Suhail
>> >>
>> >>
>>
>>
>
>
> --
> http://mixpanel.com
> Blog: http://blog.mixpanel.com
>
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
Re: Understand how to provision nodes and use cassandra in the
production
Posted by Suhail Doshi <di...@gmail.com>.
Another thing I am unsure about is my iostat:
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
avgqu-sz await svctm %util
sda1 28.00 0.00 515.00 0.00 7.39 0.00 29.39
40.18 75.43 1.94 100.00
sda2 11.00 0.00 11.00 0.00 0.09 0.00 16.00
1.08 97.82 38.55 42.40
Pretty similar on both nodes but I've seen it read more data faster than
that. Shouldn't it be reading from memory anyway?
Suhail
On Sat, Jan 30, 2010 at 3:52 PM, Suhail Doshi <di...@gmail.com>wrote:
> Also I am seeing a read latency between 73-103--assuming that's ms--seems
> high.
>
> Suhail
>
>
> On Sat, Jan 30, 2010 at 3:46 PM, Suhail Doshi <di...@gmail.com>wrote:
>
>> According jconsole on the main table I am having issues with:
>>
>> Capacity: 1164790
>> HitRate: .54
>> Size: 99753
>>
>> Right now my KeysCachedFraction is 0.2. The current memory allocated is
>> 3G. What's a suggested KeysCachedFraction value?
>>
>> Suhail
>>
>>
>> On Sat, Jan 30, 2010 at 5:58 AM, Jonathan Ellis <jb...@gmail.com>wrote:
>>
>>> the thing that will help most in 0.5 is to increase your
>>> KeysCachedFraction to 0.2 or even more, depending on your workload.
>>>
>>> On Sat, Jan 30, 2010 at 5:23 AM, Suhail Doshi <di...@gmail.com>
>>> wrote:
>>> > An issue I've been seeing is it's really hard to scale Cassandra with
>>> reads.
>>> > I've run top, vmstat, iostat. vmstat shows no swapping but iostat shows
>>> > heavy saturation of %util and await times over 90ms with max rMB/s of
>>> 7-8.
>>> >
>>> > I have over 7G of memory dedicated across two nodes. I am wondering
>>> what the
>>> > issue might be and how to solve this? I felt like 7 G would be enough.
>>> >
>>> > Suhail
>>> >
>>> > On Thu, Jan 28, 2010 at 7:32 PM, Ray Slakinski <ra...@mahalo.com>
>>> wrote:
>>> >
>>> >> Cassandra auto shards, so you just need to point at your cluster and
>>> >> cassandra does the rest. You should read up on different partitioners
>>> though
>>> >> before you go live in production, because its not too easy to switch
>>> once
>>> >> you make that decision.
>>> >>
>>> >> http://wiki.apache.org/cassandra/StorageConfiguration#Partitioner
>>> >>
>>> >> Ray Slakinski
>>> >> On 2010-01-28, at 7:29 PM, Suhail Doshi wrote:
>>> >>
>>> >> > Another piece I am interested in is how cassandra distributes the
>>> data
>>> >> > automatically. In MySQL you need to shard and you'd pick the shard
>>> to
>>> >> > request info from--how does that translate in cassandra?
>>> >> >
>>> >> > On Thu, Jan 28, 2010 at 7:23 PM, Suhail Doshi <su...@mixpanel.com>
>>> >> wrote:
>>> >> >
>>> >> >> We've started to use Cassandra in production and just have one node
>>> >> right
>>> >> >> now. Here's one of our ColumnFamilys:
>>> >> >>
>>> >> >> 16G Jan 28 22:28 SomeIndex-5467-Index.db
>>> >> >> 196M Jan 28 22:32 SomeIndex-5487-Index.db
>>> >> >>
>>> >> >> The first bottle neck you encounter is reads--writes are extremely
>>> fast
>>> >> even with one node.
>>> >> >>
>>> >> >> My question is, is the size of the *-Index.db files the amount of
>>> RAM
>>> >> you need available for Cassandra to do reads fast?
>>> >> >>
>>> >> >> What are some configuration options you would need to tweak besides
>>> the
>>> >> JVM's max memory size being larger. Is there any default
>>> configurations
>>> >> commonly missed?
>>> >> >>
>>> >> >> Next, if you provision more nodes will Cassandra distribute the
>>> data in
>>> >> memory so I don't need a single 16 GB node? Is there anything I need
>>> to
>>> >> build in my application logic to make this work correctly. Ideally, if
>>> I had
>>> >> a 16 GB index, I'd want it spread across 4 4GB nodes. Can any client
>>> connect
>>> >> to any one node request info and it will get the info back from a node
>>> that
>>> >> has that part of the index in memory?
>>> >> >>
>>> >> >> What's the best way to do efficient reads?
>>> >> >>
>>> >> >> Suhail
>>> >> >>
>>> >> >>
>>> >>
>>> >>
>>> >
>>> >
>>> > --
>>> > http://mixpanel.com
>>> > Blog: http://blog.mixpanel.com
>>> >
>>>
>>
>>
>>
>> --
>> http://mixpanel.com
>> Blog: http://blog.mixpanel.com
>>
>
>
>
> --
> http://mixpanel.com
> Blog: http://blog.mixpanel.com
>
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
Re: Understand how to provision nodes and use cassandra in the
production
Posted by Suhail Doshi <di...@gmail.com>.
Also I am seeing a read latency between 73-103--assuming that's ms--seems
high.
Suhail
On Sat, Jan 30, 2010 at 3:46 PM, Suhail Doshi <di...@gmail.com>wrote:
> According jconsole on the main table I am having issues with:
>
> Capacity: 1164790
> HitRate: .54
> Size: 99753
>
> Right now my KeysCachedFraction is 0.2. The current memory allocated is 3G.
> What's a suggested KeysCachedFraction value?
>
> Suhail
>
>
> On Sat, Jan 30, 2010 at 5:58 AM, Jonathan Ellis <jb...@gmail.com> wrote:
>
>> the thing that will help most in 0.5 is to increase your
>> KeysCachedFraction to 0.2 or even more, depending on your workload.
>>
>> On Sat, Jan 30, 2010 at 5:23 AM, Suhail Doshi <di...@gmail.com>
>> wrote:
>> > An issue I've been seeing is it's really hard to scale Cassandra with
>> reads.
>> > I've run top, vmstat, iostat. vmstat shows no swapping but iostat shows
>> > heavy saturation of %util and await times over 90ms with max rMB/s of
>> 7-8.
>> >
>> > I have over 7G of memory dedicated across two nodes. I am wondering what
>> the
>> > issue might be and how to solve this? I felt like 7 G would be enough.
>> >
>> > Suhail
>> >
>> > On Thu, Jan 28, 2010 at 7:32 PM, Ray Slakinski <ra...@mahalo.com> wrote:
>> >
>> >> Cassandra auto shards, so you just need to point at your cluster and
>> >> cassandra does the rest. You should read up on different partitioners
>> though
>> >> before you go live in production, because its not too easy to switch
>> once
>> >> you make that decision.
>> >>
>> >> http://wiki.apache.org/cassandra/StorageConfiguration#Partitioner
>> >>
>> >> Ray Slakinski
>> >> On 2010-01-28, at 7:29 PM, Suhail Doshi wrote:
>> >>
>> >> > Another piece I am interested in is how cassandra distributes the
>> data
>> >> > automatically. In MySQL you need to shard and you'd pick the shard to
>> >> > request info from--how does that translate in cassandra?
>> >> >
>> >> > On Thu, Jan 28, 2010 at 7:23 PM, Suhail Doshi <su...@mixpanel.com>
>> >> wrote:
>> >> >
>> >> >> We've started to use Cassandra in production and just have one node
>> >> right
>> >> >> now. Here's one of our ColumnFamilys:
>> >> >>
>> >> >> 16G Jan 28 22:28 SomeIndex-5467-Index.db
>> >> >> 196M Jan 28 22:32 SomeIndex-5487-Index.db
>> >> >>
>> >> >> The first bottle neck you encounter is reads--writes are extremely
>> fast
>> >> even with one node.
>> >> >>
>> >> >> My question is, is the size of the *-Index.db files the amount of
>> RAM
>> >> you need available for Cassandra to do reads fast?
>> >> >>
>> >> >> What are some configuration options you would need to tweak besides
>> the
>> >> JVM's max memory size being larger. Is there any default configurations
>> >> commonly missed?
>> >> >>
>> >> >> Next, if you provision more nodes will Cassandra distribute the data
>> in
>> >> memory so I don't need a single 16 GB node? Is there anything I need to
>> >> build in my application logic to make this work correctly. Ideally, if
>> I had
>> >> a 16 GB index, I'd want it spread across 4 4GB nodes. Can any client
>> connect
>> >> to any one node request info and it will get the info back from a node
>> that
>> >> has that part of the index in memory?
>> >> >>
>> >> >> What's the best way to do efficient reads?
>> >> >>
>> >> >> Suhail
>> >> >>
>> >> >>
>> >>
>> >>
>> >
>> >
>> > --
>> > http://mixpanel.com
>> > Blog: http://blog.mixpanel.com
>> >
>>
>
>
>
> --
> http://mixpanel.com
> Blog: http://blog.mixpanel.com
>
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
Re: Understand how to provision nodes and use cassandra in the
production
Posted by Suhail Doshi <di...@gmail.com>.
Haven't seen much success between 0.1, 0.2, 0.3
On Sat, Jan 30, 2010 at 4:04 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> you want to get hitrate to 0.9 or so, i.e. 90% of index lookups don't
> have to hit disk. play with KCF and see what happens. and use
> jconsole to see how close you are getting to your 3GB limit (hit the
> GC button to see how much memory is "really" being used, and then add
> 25% or so for a reasonable padding).
>
> On Sat, Jan 30, 2010 at 5:46 PM, Suhail Doshi <di...@gmail.com>
> wrote:
> > According jconsole on the main table I am having issues with:
> >
> > Capacity: 1164790
> > HitRate: .54
> > Size: 99753
> >
> > Right now my KeysCachedFraction is 0.2. The current memory allocated is
> 3G.
> > What's a suggested KeysCachedFraction value?
> >
> > Suhail
> >
> > On Sat, Jan 30, 2010 at 5:58 AM, Jonathan Ellis <jb...@gmail.com>
> wrote:
> >
> >> the thing that will help most in 0.5 is to increase your
> >> KeysCachedFraction to 0.2 or even more, depending on your workload.
> >>
> >> On Sat, Jan 30, 2010 at 5:23 AM, Suhail Doshi <digitalwarfare@gmail.com
> >
> >> wrote:
> >> > An issue I've been seeing is it's really hard to scale Cassandra with
> >> reads.
> >> > I've run top, vmstat, iostat. vmstat shows no swapping but iostat
> shows
> >> > heavy saturation of %util and await times over 90ms with max rMB/s of
> >> 7-8.
> >> >
> >> > I have over 7G of memory dedicated across two nodes. I am wondering
> what
> >> the
> >> > issue might be and how to solve this? I felt like 7 G would be enough.
> >> >
> >> > Suhail
> >> >
> >> > On Thu, Jan 28, 2010 at 7:32 PM, Ray Slakinski <ra...@mahalo.com>
> wrote:
> >> >
> >> >> Cassandra auto shards, so you just need to point at your cluster and
> >> >> cassandra does the rest. You should read up on different partitioners
> >> though
> >> >> before you go live in production, because its not too easy to switch
> >> once
> >> >> you make that decision.
> >> >>
> >> >> http://wiki.apache.org/cassandra/StorageConfiguration#Partitioner
> >> >>
> >> >> Ray Slakinski
> >> >> On 2010-01-28, at 7:29 PM, Suhail Doshi wrote:
> >> >>
> >> >> > Another piece I am interested in is how cassandra distributes the
> data
> >> >> > automatically. In MySQL you need to shard and you'd pick the shard
> to
> >> >> > request info from--how does that translate in cassandra?
> >> >> >
> >> >> > On Thu, Jan 28, 2010 at 7:23 PM, Suhail Doshi <suhail@mixpanel.com
> >
> >> >> wrote:
> >> >> >
> >> >> >> We've started to use Cassandra in production and just have one
> node
> >> >> right
> >> >> >> now. Here's one of our ColumnFamilys:
> >> >> >>
> >> >> >> 16G Jan 28 22:28 SomeIndex-5467-Index.db
> >> >> >> 196M Jan 28 22:32 SomeIndex-5487-Index.db
> >> >> >>
> >> >> >> The first bottle neck you encounter is reads--writes are extremely
> >> fast
> >> >> even with one node.
> >> >> >>
> >> >> >> My question is, is the size of the *-Index.db files the amount of
> RAM
> >> >> you need available for Cassandra to do reads fast?
> >> >> >>
> >> >> >> What are some configuration options you would need to tweak
> besides
> >> the
> >> >> JVM's max memory size being larger. Is there any default
> configurations
> >> >> commonly missed?
> >> >> >>
> >> >> >> Next, if you provision more nodes will Cassandra distribute the
> data
> >> in
> >> >> memory so I don't need a single 16 GB node? Is there anything I need
> to
> >> >> build in my application logic to make this work correctly. Ideally,
> if I
> >> had
> >> >> a 16 GB index, I'd want it spread across 4 4GB nodes. Can any client
> >> connect
> >> >> to any one node request info and it will get the info back from a
> node
> >> that
> >> >> has that part of the index in memory?
> >> >> >>
> >> >> >> What's the best way to do efficient reads?
> >> >> >>
> >> >> >> Suhail
> >> >> >>
> >> >> >>
> >> >>
> >> >>
> >> >
> >> >
> >> > --
> >> > http://mixpanel.com
> >> > Blog: http://blog.mixpanel.com
> >> >
> >>
> >
> >
> >
> > --
> > http://mixpanel.com
> > Blog: http://blog.mixpanel.com
> >
>
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
Re: Understand how to provision nodes and use cassandra in the
production
Posted by Jonathan Ellis <jb...@gmail.com>.
you want to get hitrate to 0.9 or so, i.e. 90% of index lookups don't
have to hit disk. play with KCF and see what happens. and use
jconsole to see how close you are getting to your 3GB limit (hit the
GC button to see how much memory is "really" being used, and then add
25% or so for a reasonable padding).
On Sat, Jan 30, 2010 at 5:46 PM, Suhail Doshi <di...@gmail.com> wrote:
> According jconsole on the main table I am having issues with:
>
> Capacity: 1164790
> HitRate: .54
> Size: 99753
>
> Right now my KeysCachedFraction is 0.2. The current memory allocated is 3G.
> What's a suggested KeysCachedFraction value?
>
> Suhail
>
> On Sat, Jan 30, 2010 at 5:58 AM, Jonathan Ellis <jb...@gmail.com> wrote:
>
>> the thing that will help most in 0.5 is to increase your
>> KeysCachedFraction to 0.2 or even more, depending on your workload.
>>
>> On Sat, Jan 30, 2010 at 5:23 AM, Suhail Doshi <di...@gmail.com>
>> wrote:
>> > An issue I've been seeing is it's really hard to scale Cassandra with
>> reads.
>> > I've run top, vmstat, iostat. vmstat shows no swapping but iostat shows
>> > heavy saturation of %util and await times over 90ms with max rMB/s of
>> 7-8.
>> >
>> > I have over 7G of memory dedicated across two nodes. I am wondering what
>> the
>> > issue might be and how to solve this? I felt like 7 G would be enough.
>> >
>> > Suhail
>> >
>> > On Thu, Jan 28, 2010 at 7:32 PM, Ray Slakinski <ra...@mahalo.com> wrote:
>> >
>> >> Cassandra auto shards, so you just need to point at your cluster and
>> >> cassandra does the rest. You should read up on different partitioners
>> though
>> >> before you go live in production, because its not too easy to switch
>> once
>> >> you make that decision.
>> >>
>> >> http://wiki.apache.org/cassandra/StorageConfiguration#Partitioner
>> >>
>> >> Ray Slakinski
>> >> On 2010-01-28, at 7:29 PM, Suhail Doshi wrote:
>> >>
>> >> > Another piece I am interested in is how cassandra distributes the data
>> >> > automatically. In MySQL you need to shard and you'd pick the shard to
>> >> > request info from--how does that translate in cassandra?
>> >> >
>> >> > On Thu, Jan 28, 2010 at 7:23 PM, Suhail Doshi <su...@mixpanel.com>
>> >> wrote:
>> >> >
>> >> >> We've started to use Cassandra in production and just have one node
>> >> right
>> >> >> now. Here's one of our ColumnFamilys:
>> >> >>
>> >> >> 16G Jan 28 22:28 SomeIndex-5467-Index.db
>> >> >> 196M Jan 28 22:32 SomeIndex-5487-Index.db
>> >> >>
>> >> >> The first bottle neck you encounter is reads--writes are extremely
>> fast
>> >> even with one node.
>> >> >>
>> >> >> My question is, is the size of the *-Index.db files the amount of RAM
>> >> you need available for Cassandra to do reads fast?
>> >> >>
>> >> >> What are some configuration options you would need to tweak besides
>> the
>> >> JVM's max memory size being larger. Is there any default configurations
>> >> commonly missed?
>> >> >>
>> >> >> Next, if you provision more nodes will Cassandra distribute the data
>> in
>> >> memory so I don't need a single 16 GB node? Is there anything I need to
>> >> build in my application logic to make this work correctly. Ideally, if I
>> had
>> >> a 16 GB index, I'd want it spread across 4 4GB nodes. Can any client
>> connect
>> >> to any one node request info and it will get the info back from a node
>> that
>> >> has that part of the index in memory?
>> >> >>
>> >> >> What's the best way to do efficient reads?
>> >> >>
>> >> >> Suhail
>> >> >>
>> >> >>
>> >>
>> >>
>> >
>> >
>> > --
>> > http://mixpanel.com
>> > Blog: http://blog.mixpanel.com
>> >
>>
>
>
>
> --
> http://mixpanel.com
> Blog: http://blog.mixpanel.com
>
Re: Understand how to provision nodes and use cassandra in the
production
Posted by Suhail Doshi <di...@gmail.com>.
According jconsole on the main table I am having issues with:
Capacity: 1164790
HitRate: .54
Size: 99753
Right now my KeysCachedFraction is 0.2. The current memory allocated is 3G.
What's a suggested KeysCachedFraction value?
Suhail
On Sat, Jan 30, 2010 at 5:58 AM, Jonathan Ellis <jb...@gmail.com> wrote:
> the thing that will help most in 0.5 is to increase your
> KeysCachedFraction to 0.2 or even more, depending on your workload.
>
> On Sat, Jan 30, 2010 at 5:23 AM, Suhail Doshi <di...@gmail.com>
> wrote:
> > An issue I've been seeing is it's really hard to scale Cassandra with
> reads.
> > I've run top, vmstat, iostat. vmstat shows no swapping but iostat shows
> > heavy saturation of %util and await times over 90ms with max rMB/s of
> 7-8.
> >
> > I have over 7G of memory dedicated across two nodes. I am wondering what
> the
> > issue might be and how to solve this? I felt like 7 G would be enough.
> >
> > Suhail
> >
> > On Thu, Jan 28, 2010 at 7:32 PM, Ray Slakinski <ra...@mahalo.com> wrote:
> >
> >> Cassandra auto shards, so you just need to point at your cluster and
> >> cassandra does the rest. You should read up on different partitioners
> though
> >> before you go live in production, because its not too easy to switch
> once
> >> you make that decision.
> >>
> >> http://wiki.apache.org/cassandra/StorageConfiguration#Partitioner
> >>
> >> Ray Slakinski
> >> On 2010-01-28, at 7:29 PM, Suhail Doshi wrote:
> >>
> >> > Another piece I am interested in is how cassandra distributes the data
> >> > automatically. In MySQL you need to shard and you'd pick the shard to
> >> > request info from--how does that translate in cassandra?
> >> >
> >> > On Thu, Jan 28, 2010 at 7:23 PM, Suhail Doshi <su...@mixpanel.com>
> >> wrote:
> >> >
> >> >> We've started to use Cassandra in production and just have one node
> >> right
> >> >> now. Here's one of our ColumnFamilys:
> >> >>
> >> >> 16G Jan 28 22:28 SomeIndex-5467-Index.db
> >> >> 196M Jan 28 22:32 SomeIndex-5487-Index.db
> >> >>
> >> >> The first bottle neck you encounter is reads--writes are extremely
> fast
> >> even with one node.
> >> >>
> >> >> My question is, is the size of the *-Index.db files the amount of RAM
> >> you need available for Cassandra to do reads fast?
> >> >>
> >> >> What are some configuration options you would need to tweak besides
> the
> >> JVM's max memory size being larger. Is there any default configurations
> >> commonly missed?
> >> >>
> >> >> Next, if you provision more nodes will Cassandra distribute the data
> in
> >> memory so I don't need a single 16 GB node? Is there anything I need to
> >> build in my application logic to make this work correctly. Ideally, if I
> had
> >> a 16 GB index, I'd want it spread across 4 4GB nodes. Can any client
> connect
> >> to any one node request info and it will get the info back from a node
> that
> >> has that part of the index in memory?
> >> >>
> >> >> What's the best way to do efficient reads?
> >> >>
> >> >> Suhail
> >> >>
> >> >>
> >>
> >>
> >
> >
> > --
> > http://mixpanel.com
> > Blog: http://blog.mixpanel.com
> >
>
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
Re: Understand how to provision nodes and use cassandra in the
production
Posted by Jonathan Ellis <jb...@gmail.com>.
the thing that will help most in 0.5 is to increase your
KeysCachedFraction to 0.2 or even more, depending on your workload.
On Sat, Jan 30, 2010 at 5:23 AM, Suhail Doshi <di...@gmail.com> wrote:
> An issue I've been seeing is it's really hard to scale Cassandra with reads.
> I've run top, vmstat, iostat. vmstat shows no swapping but iostat shows
> heavy saturation of %util and await times over 90ms with max rMB/s of 7-8.
>
> I have over 7G of memory dedicated across two nodes. I am wondering what the
> issue might be and how to solve this? I felt like 7 G would be enough.
>
> Suhail
>
> On Thu, Jan 28, 2010 at 7:32 PM, Ray Slakinski <ra...@mahalo.com> wrote:
>
>> Cassandra auto shards, so you just need to point at your cluster and
>> cassandra does the rest. You should read up on different partitioners though
>> before you go live in production, because its not too easy to switch once
>> you make that decision.
>>
>> http://wiki.apache.org/cassandra/StorageConfiguration#Partitioner
>>
>> Ray Slakinski
>> On 2010-01-28, at 7:29 PM, Suhail Doshi wrote:
>>
>> > Another piece I am interested in is how cassandra distributes the data
>> > automatically. In MySQL you need to shard and you'd pick the shard to
>> > request info from--how does that translate in cassandra?
>> >
>> > On Thu, Jan 28, 2010 at 7:23 PM, Suhail Doshi <su...@mixpanel.com>
>> wrote:
>> >
>> >> We've started to use Cassandra in production and just have one node
>> right
>> >> now. Here's one of our ColumnFamilys:
>> >>
>> >> 16G Jan 28 22:28 SomeIndex-5467-Index.db
>> >> 196M Jan 28 22:32 SomeIndex-5487-Index.db
>> >>
>> >> The first bottle neck you encounter is reads--writes are extremely fast
>> even with one node.
>> >>
>> >> My question is, is the size of the *-Index.db files the amount of RAM
>> you need available for Cassandra to do reads fast?
>> >>
>> >> What are some configuration options you would need to tweak besides the
>> JVM's max memory size being larger. Is there any default configurations
>> commonly missed?
>> >>
>> >> Next, if you provision more nodes will Cassandra distribute the data in
>> memory so I don't need a single 16 GB node? Is there anything I need to
>> build in my application logic to make this work correctly. Ideally, if I had
>> a 16 GB index, I'd want it spread across 4 4GB nodes. Can any client connect
>> to any one node request info and it will get the info back from a node that
>> has that part of the index in memory?
>> >>
>> >> What's the best way to do efficient reads?
>> >>
>> >> Suhail
>> >>
>> >>
>>
>>
>
>
> --
> http://mixpanel.com
> Blog: http://blog.mixpanel.com
>
Re: Understand how to provision nodes and use cassandra in the
production
Posted by Suhail Doshi <di...@gmail.com>.
An issue I've been seeing is it's really hard to scale Cassandra with reads.
I've run top, vmstat, iostat. vmstat shows no swapping but iostat shows
heavy saturation of %util and await times over 90ms with max rMB/s of 7-8.
I have over 7G of memory dedicated across two nodes. I am wondering what the
issue might be and how to solve this? I felt like 7 G would be enough.
Suhail
On Thu, Jan 28, 2010 at 7:32 PM, Ray Slakinski <ra...@mahalo.com> wrote:
> Cassandra auto shards, so you just need to point at your cluster and
> cassandra does the rest. You should read up on different partitioners though
> before you go live in production, because its not too easy to switch once
> you make that decision.
>
> http://wiki.apache.org/cassandra/StorageConfiguration#Partitioner
>
> Ray Slakinski
> On 2010-01-28, at 7:29 PM, Suhail Doshi wrote:
>
> > Another piece I am interested in is how cassandra distributes the data
> > automatically. In MySQL you need to shard and you'd pick the shard to
> > request info from--how does that translate in cassandra?
> >
> > On Thu, Jan 28, 2010 at 7:23 PM, Suhail Doshi <su...@mixpanel.com>
> wrote:
> >
> >> We've started to use Cassandra in production and just have one node
> right
> >> now. Here's one of our ColumnFamilys:
> >>
> >> 16G Jan 28 22:28 SomeIndex-5467-Index.db
> >> 196M Jan 28 22:32 SomeIndex-5487-Index.db
> >>
> >> The first bottle neck you encounter is reads--writes are extremely fast
> even with one node.
> >>
> >> My question is, is the size of the *-Index.db files the amount of RAM
> you need available for Cassandra to do reads fast?
> >>
> >> What are some configuration options you would need to tweak besides the
> JVM's max memory size being larger. Is there any default configurations
> commonly missed?
> >>
> >> Next, if you provision more nodes will Cassandra distribute the data in
> memory so I don't need a single 16 GB node? Is there anything I need to
> build in my application logic to make this work correctly. Ideally, if I had
> a 16 GB index, I'd want it spread across 4 4GB nodes. Can any client connect
> to any one node request info and it will get the info back from a node that
> has that part of the index in memory?
> >>
> >> What's the best way to do efficient reads?
> >>
> >> Suhail
> >>
> >>
>
>
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
Re: Understand how to provision nodes and use cassandra in the production
Posted by Ray Slakinski <ra...@mahalo.com>.
Cassandra auto shards, so you just need to point at your cluster and cassandra does the rest. You should read up on different partitioners though before you go live in production, because its not too easy to switch once you make that decision.
http://wiki.apache.org/cassandra/StorageConfiguration#Partitioner
Ray Slakinski
On 2010-01-28, at 7:29 PM, Suhail Doshi wrote:
> Another piece I am interested in is how cassandra distributes the data
> automatically. In MySQL you need to shard and you'd pick the shard to
> request info from--how does that translate in cassandra?
>
> On Thu, Jan 28, 2010 at 7:23 PM, Suhail Doshi <su...@mixpanel.com> wrote:
>
>> We've started to use Cassandra in production and just have one node right
>> now. Here's one of our ColumnFamilys:
>>
>> 16G Jan 28 22:28 SomeIndex-5467-Index.db
>> 196M Jan 28 22:32 SomeIndex-5487-Index.db
>>
>> The first bottle neck you encounter is reads--writes are extremely fast even with one node.
>>
>> My question is, is the size of the *-Index.db files the amount of RAM you need available for Cassandra to do reads fast?
>>
>> What are some configuration options you would need to tweak besides the JVM's max memory size being larger. Is there any default configurations commonly missed?
>>
>> Next, if you provision more nodes will Cassandra distribute the data in memory so I don't need a single 16 GB node? Is there anything I need to build in my application logic to make this work correctly. Ideally, if I had a 16 GB index, I'd want it spread across 4 4GB nodes. Can any client connect to any one node request info and it will get the info back from a node that has that part of the index in memory?
>>
>> What's the best way to do efficient reads?
>>
>> Suhail
>>
>>
Re: Understand how to provision nodes and use cassandra in the
production
Posted by Suhail Doshi <su...@mixpanel.com>.
Another piece I am interested in is how cassandra distributes the data
automatically. In MySQL you need to shard and you'd pick the shard to
request info from--how does that translate in cassandra?
On Thu, Jan 28, 2010 at 7:23 PM, Suhail Doshi <su...@mixpanel.com> wrote:
> We've started to use Cassandra in production and just have one node right
> now. Here's one of our ColumnFamilys:
>
> 16G Jan 28 22:28 SomeIndex-5467-Index.db
> 196M Jan 28 22:32 SomeIndex-5487-Index.db
>
> The first bottle neck you encounter is reads--writes are extremely fast even with one node.
>
> My question is, is the size of the *-Index.db files the amount of RAM you need available for Cassandra to do reads fast?
>
> What are some configuration options you would need to tweak besides the JVM's max memory size being larger. Is there any default configurations commonly missed?
>
> Next, if you provision more nodes will Cassandra distribute the data in memory so I don't need a single 16 GB node? Is there anything I need to build in my application logic to make this work correctly. Ideally, if I had a 16 GB index, I'd want it spread across 4 4GB nodes. Can any client connect to any one node request info and it will get the info back from a node that has that part of the index in memory?
>
> What's the best way to do efficient reads?
>
> Suhail
>
>
Re: Understand how to provision nodes and use cassandra in the
production
Posted by Jonathan Ellis <jb...@gmail.com>.
On Thu, Jan 28, 2010 at 9:23 PM, Suhail Doshi <su...@mixpanel.com> wrote:
> We've started to use Cassandra in production and just have one node right
> now. Here's one of our ColumnFamilys:
>
>
> 16G Jan 28 22:28 SomeIndex-5467-Index.db
> 196M Jan 28 22:32 SomeIndex-5487-Index.db
>
> The first bottle neck you encounter is reads--writes are extremely
> fast even with one node.
>
> My question is, is the size of the *-Index.db files the amount of RAM
> you need available for Cassandra to do reads fast?
No. It depends how much of your data is "hot." IIRC you are running
trunk -- look at the key cache hit rate with various KeyCacheFractions
and see how large it has to be to get an 80% hit rate or so.
> Next, if you provision more nodes will Cassandra distribute the data
> in memory so I don't need a single 16 GB node?
Yes. See the Ring Management section here:
http://wiki.apache.org/cassandra/Operations
> Can any client connect to any one node request info and it will
> get the info back from a node that has that part of the index in
> memory?
Yes.