How can I scale my read rate?

[S G <sg...@gmail.com>]

Hi,

I have a 9 node cassandra cluster, each node has:
RAM: 5.5gb
Disk: 64gb
C* version: 3.3
Java: 1.8.0_51

The cluster stores about 2 million rows and partition keys for all these
rows are unique.

I am stress-reading it from a 12-node client cluster.
Each read-client has 50 threads, so total 600 read-threads from 12 machines.
Each read query is get-by-primary-key where keys are read randomly from a
file having all primary-keys.


I am able to get only 15,000 reads/second from the entire system.
Is this the best read performance I can expect?
Are there any benchmarks I can compare against?


I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching to
{'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.

Thx,
SG

Re: How can I scale my read rate?

[benjamin roth <br...@gmail.com>]

Maybe lower the compression chunk size and give the LCS a try

Am 15.03.2017 1:22 vorm. schrieb "D. Salvatore" <dd...@gmail.com>:

> Hi,
> What is roughly the average size of your rows? Are you asking all the row
> or just some fields?
> Did you check the disk utilisation?
>
> On my testbed composed by 4 VMs, each with 8 CPUs and 4GB of RAM, I
> roughly get 14500 read/sec.
>
> Salvatore
>
> 2017-03-14 22:45 GMT+00:00 S G <sg...@gmail.com>:
>
>> Hi,
>>
>> I am trying to scale my read throughput.
>>
>> I have a 12 node Cassandra cluster, each node has:
>> RAM: 5.5gb
>> Disk: 64gb
>> C* version: 3.3
>> Java: 1.8.0_51
>>
>> The cluster has 2 tables - each with 2 million rows.
>> Partition keys for all these rows are unique.
>>
>> I am stress-reading it from a 15-node client cluster.
>> Each read-client has 40 threads, so total 600 read-threads from 12
>> machines.
>> Each read query is get-by-primary-key where keys are read randomly from a
>> file having all primary-keys.
>>
>>
>> I am able to get only 15,000 reads/second from the entire system.
>> Is this the best read performance I can expect?
>> Are there any benchmarks I can compare against?
>>
>>
>> I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching to
>> {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
>>
>> I have also captured few JMX metric graphs as follows:
>>
>> ​
>> Garbage collection metrics did not look very interesting, hence I have
>> not added the same.
>>
>> Any ideas what more I can do to debug the system and increase its read
>> throughput?
>>
>> Thanks
>> SG
>>
>>
>>
>> On Sat, Mar 11, 2017 at 2:33 PM, Jeff Jirsa <jj...@gmail.com> wrote:
>>
>>> 5.5G of ram isn't very much for a jvm based database. Having fewer
>>> instances with more ram will likely give you better performance.
>>>
>>> Also, this is probably a better question for the user list than the Dev
>>> list
>>>
>>> --
>>> Jeff Jirsa
>>>
>>>
>>> > On Mar 11, 2017, at 1:20 PM, S G <sg...@gmail.com> wrote:
>>> >
>>> > Hi,
>>> >
>>> > I have a 9 node cassandra cluster, each node has:
>>> > RAM: 5.5gb
>>> > Disk: 64gb
>>> > C* version: 3.3
>>> > Java: 1.8.0_51
>>> >
>>> > The cluster stores about 2 million rows and partition keys for all
>>> these
>>> > rows are unique.
>>> >
>>> > I am stress-reading it from a 12-node client cluster.
>>> > Each read-client has 50 threads, so total 600 read-threads from 12
>>> machines.
>>> > Each read query is get-by-primary-key where keys are read randomly
>>> from a
>>> > file having all primary-keys.
>>> >
>>> >
>>> > I am able to get only 15,000 reads/second from the entire system.
>>> > Is this the best read performance I can expect?
>>> > Are there any benchmarks I can compare against?
>>> >
>>> >
>>> > I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching
>>> to
>>> > {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
>>> >
>>> > Thx,
>>> > SG
>>>
>>
>>
>

Re: How can I scale my read rate?

["D. Salvatore" <dd...@gmail.com>]

Hi,
What is roughly the average size of your rows? Are you asking all the row
or just some fields?
Did you check the disk utilisation?

On my testbed composed by 4 VMs, each with 8 CPUs and 4GB of RAM, I roughly
get 14500 read/sec.

Salvatore

2017-03-14 22:45 GMT+00:00 S G <sg...@gmail.com>:

> Hi,
>
> I am trying to scale my read throughput.
>
> I have a 12 node Cassandra cluster, each node has:
> RAM: 5.5gb
> Disk: 64gb
> C* version: 3.3
> Java: 1.8.0_51
>
> The cluster has 2 tables - each with 2 million rows.
> Partition keys for all these rows are unique.
>
> I am stress-reading it from a 15-node client cluster.
> Each read-client has 40 threads, so total 600 read-threads from 12
> machines.
> Each read query is get-by-primary-key where keys are read randomly from a
> file having all primary-keys.
>
>
> I am able to get only 15,000 reads/second from the entire system.
> Is this the best read performance I can expect?
> Are there any benchmarks I can compare against?
>
>
> I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching to
> {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
>
> I have also captured few JMX metric graphs as follows:
>
> ​
> Garbage collection metrics did not look very interesting, hence I have not
> added the same.
>
> Any ideas what more I can do to debug the system and increase its read
> throughput?
>
> Thanks
> SG
>
>
>
> On Sat, Mar 11, 2017 at 2:33 PM, Jeff Jirsa <jj...@gmail.com> wrote:
>
>> 5.5G of ram isn't very much for a jvm based database. Having fewer
>> instances with more ram will likely give you better performance.
>>
>> Also, this is probably a better question for the user list than the Dev
>> list
>>
>> --
>> Jeff Jirsa
>>
>>
>> > On Mar 11, 2017, at 1:20 PM, S G <sg...@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > I have a 9 node cassandra cluster, each node has:
>> > RAM: 5.5gb
>> > Disk: 64gb
>> > C* version: 3.3
>> > Java: 1.8.0_51
>> >
>> > The cluster stores about 2 million rows and partition keys for all these
>> > rows are unique.
>> >
>> > I am stress-reading it from a 12-node client cluster.
>> > Each read-client has 50 threads, so total 600 read-threads from 12
>> machines.
>> > Each read query is get-by-primary-key where keys are read randomly from
>> a
>> > file having all primary-keys.
>> >
>> >
>> > I am able to get only 15,000 reads/second from the entire system.
>> > Is this the best read performance I can expect?
>> > Are there any benchmarks I can compare against?
>> >
>> >
>> > I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching
>> to
>> > {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
>> >
>> > Thx,
>> > SG
>>
>
>

Re: How can I scale my read rate?

[S G <sg...@gmail.com>]

Hi Benjamin,

When I moved from 2cores/node to 8cores/node - I was able to increase
throughput from 15k/s to 40k/s
I increased cores for both Cassandra and the read-stress-client.

Then I tried the following and each had no effect:

1) Use G1 collector on Cassandra (-XX:+UseG1GC -XX:MaxGCPauseMillis=200)
2) Increase nodes in Cassandra cluster from 12 to 16
3) Increase nodes in Client cluster from 16 to 22
4) Increase threads in Client cluster from 960 to 13200
5) Increase concurrent_reads in cassandra.yaml from 32 to 128
6) Increase native_transport_max_threads in cassandra.yaml from 128 to 512
7) Gave more memory to Client cluster from 2G to 4G and then to 8G

I will look at the nodetool gcstats and see if something is interesting
there.

Thanks
SG

On Fri, Mar 17, 2017 at 11:03 AM, benjamin roth <br...@gmail.com> wrote:

> Looks like you have a concurrency problem.
> A 99% 0.3ms read time looks really decent. But if you have 12 nodes and no
> concurrency, each node could process 3333 req/s of these resulting in 40000
> req/s for 12 nodes.
> That even WITHOUT concurrency. With 8 cores you can get out much more out
> of it. Of course this is a very rough estimation but should show that 15k/s
> is still quite low.
> It seems that something is blocking. As can be seen in max latency. You
> should take a look at GC related JMX metrics (or nodetool gcstats).
> And maybe you should try to tweak concurrency settings in cassandra.yaml
> like concurrent_reads. The defaults are rather conservative.
>
> 2017-03-17 17:26 GMT+01:00 S G <sg...@gmail.com>:
>
>> Thank you all for the nice pointers.
>> I am using 22 machines each running 60 threads as reading clients.
>> Each client has 8 cores and 23 GB RAM.
>>
>> On the Cassandra side, I bumped up the machines from 2 core, 5.5gb nodes
>> to 8 core, 24 gb nodes and the performance improved from 15,000 to 35,000
>> reads/second.
>>
>> Now it is stuck again at this throughput.
>>
>> I have tried using G1 garbage collector and played with
>> increasing/decreasing threads and machines on the clients reading Cassandra.
>> No effect.
>>
>> Strangest part is this:
>> It seems adding more nodes to Cassandra cluster has no effect !
>> I had 12 machines a while ago and when I bumped the cluster size to 16
>> nodes (33% bigger), there was zero improvement on the read performance.
>> Is the datastax client smart enough to route a get-by-primary-key query
>> to the right node always?
>> Or does the Cassandra nodes redirect queries to the right node after they
>> find a right token-range?
>> If its the latter, then it seems a bigger cluster size's advantages are
>> being offset by an increasing number of redirects between the Cassandra
>> nodes.
>>
>> I tried to look at Cassandra JMX metrics but there are way too many of
>> them.
>> What metrics should I look into?
>>
>>
>> The tablehistogram for the table being queried looks awesome.
>>
>> ​
>> How do I diagnose this system for a faulty setting?
>> What are the usual suspects in such cases?
>>
>> Thanks
>> SG
>>
>>
>>
>>
>> On Thu, Mar 16, 2017 at 8:11 AM, Patrick McFadin <pm...@gmail.com>
>> wrote:
>>
>>> My general rule of thumb when it comes to stress clients to cluster
>>> under load is around 2:1. So with a 12 node cluster, 24 stress clients. As
>>> Eric stated, monitoring your stress clients is critical to making sure you
>>> have adequate load being produced.
>>>
>>> Patrick
>>>
>>> On Wed, Mar 15, 2017 at 3:50 PM, Carlos Rolo <ro...@pythian.com> wrote:
>>>
>>>> Plus is one of the most documented questions available...
>>>>
>>>> Regards,
>>>>
>>>> Carlos Juzarte Rolo
>>>> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>>>>
>>>> Pythian - Love your data
>>>>
>>>> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
>>>> *linkedin.com/in/carlosjuzarterolo
>>>> <http://linkedin.com/in/carlosjuzarterolo>*
>>>> Mobile: +351 918 918 100 <+351%20918%20918%20100>
>>>> www.pythian.com
>>>>
>>>> On Wed, Mar 15, 2017 at 10:29 PM, James Carman <
>>>> james@carmanconsulting.com> wrote:
>>>>
>>>>> Come on, man.  That's not called for.  Someone is coming here to the
>>>>> community for help.  Let's be courteous, please.
>>>>>
>>>>> On Wed, Mar 15, 2017 at 5:12 PM daemeon reiydelle <da...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> with that brain damaged memory setting yes.
>>>>>>
>>>>>>
>>>>>> *.......*
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 <(415)%20501-0198>London
>>>>>> (+44) (0) 20 8144 9872 <+44%2020%208144%209872>*
>>>>>>
>>>>>> On Tue, Mar 14, 2017 at 3:45 PM, S G <sg...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am trying to scale my read throughput.
>>>>>>
>>>>>> I have a 12 node Cassandra cluster, each node has:
>>>>>> RAM: 5.5gb
>>>>>> Disk: 64gb
>>>>>> C* version: 3.3
>>>>>> Java: 1.8.0_51
>>>>>>
>>>>>> The cluster has 2 tables - each with 2 million rows.
>>>>>> Partition keys for all these rows are unique.
>>>>>>
>>>>>> I am stress-reading it from a 15-node client cluster.
>>>>>> Each read-client has 40 threads, so total 600 read-threads from 12
>>>>>> machines.
>>>>>> Each read query is get-by-primary-key where keys are read randomly
>>>>>> from a file having all primary-keys.
>>>>>>
>>>>>>
>>>>>> I am able to get only 15,000 reads/second from the entire system.
>>>>>> Is this the best read performance I can expect?
>>>>>> Are there any benchmarks I can compare against?
>>>>>>
>>>>>>
>>>>>> I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and
>>>>>> caching to {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no
>>>>>> effect.
>>>>>>
>>>>>> I have also captured few JMX metric graphs as follows:
>>>>>> [image: Cassandra-Bulk-Read-Metrics.png]
>>>>>> ​
>>>>>> Garbage collection metrics did not look very interesting, hence I
>>>>>> have not added the same.
>>>>>>
>>>>>> Any ideas what more I can do to debug the system and increase its
>>>>>> read throughput?
>>>>>>
>>>>>> Thanks
>>>>>> SG
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, Mar 11, 2017 at 2:33 PM, Jeff Jirsa <jj...@gmail.com> wrote:
>>>>>>
>>>>>> 5.5G of ram isn't very much for a jvm based database. Having fewer
>>>>>> instances with more ram will likely give you better performance.
>>>>>>
>>>>>> Also, this is probably a better question for the user list than the
>>>>>> Dev list
>>>>>>
>>>>>> --
>>>>>> Jeff Jirsa
>>>>>>
>>>>>>
>>>>>> > On Mar 11, 2017, at 1:20 PM, S G <sg...@gmail.com> wrote:
>>>>>> >
>>>>>> > Hi,
>>>>>> >
>>>>>> > I have a 9 node cassandra cluster, each node has:
>>>>>> > RAM: 5.5gb
>>>>>> > Disk: 64gb
>>>>>> > C* version: 3.3
>>>>>> > Java: 1.8.0_51
>>>>>> >
>>>>>> > The cluster stores about 2 million rows and partition keys for all
>>>>>> these
>>>>>> > rows are unique.
>>>>>> >
>>>>>> > I am stress-reading it from a 12-node client cluster.
>>>>>> > Each read-client has 50 threads, so total 600 read-threads from 12
>>>>>> machines.
>>>>>> > Each read query is get-by-primary-key where keys are read randomly
>>>>>> from a
>>>>>> > file having all primary-keys.
>>>>>> >
>>>>>> >
>>>>>> > I am able to get only 15,000 reads/second from the entire system.
>>>>>> > Is this the best read performance I can expect?
>>>>>> > Are there any benchmarks I can compare against?
>>>>>> >
>>>>>> >
>>>>>> > I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and
>>>>>> caching to
>>>>>> > {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
>>>>>> >
>>>>>> > Thx,
>>>>>> > SG
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>> --
>>>>
>>>>
>>>>
>>>>
>>>
>>
>

Re: How can I scale my read rate?

[benjamin roth <br...@gmail.com>]

Looks like you have a concurrency problem.
A 99% 0.3ms read time looks really decent. But if you have 12 nodes and no
concurrency, each node could process 3333 req/s of these resulting in 40000
req/s for 12 nodes.
That even WITHOUT concurrency. With 8 cores you can get out much more out
of it. Of course this is a very rough estimation but should show that 15k/s
is still quite low.
It seems that something is blocking. As can be seen in max latency. You
should take a look at GC related JMX metrics (or nodetool gcstats).
And maybe you should try to tweak concurrency settings in cassandra.yaml
like concurrent_reads. The defaults are rather conservative.

2017-03-17 17:26 GMT+01:00 S G <sg...@gmail.com>:

> Thank you all for the nice pointers.
> I am using 22 machines each running 60 threads as reading clients.
> Each client has 8 cores and 23 GB RAM.
>
> On the Cassandra side, I bumped up the machines from 2 core, 5.5gb nodes
> to 8 core, 24 gb nodes and the performance improved from 15,000 to 35,000
> reads/second.
>
> Now it is stuck again at this throughput.
>
> I have tried using G1 garbage collector and played with
> increasing/decreasing threads and machines on the clients reading Cassandra.
> No effect.
>
> Strangest part is this:
> It seems adding more nodes to Cassandra cluster has no effect !
> I had 12 machines a while ago and when I bumped the cluster size to 16
> nodes (33% bigger), there was zero improvement on the read performance.
> Is the datastax client smart enough to route a get-by-primary-key query to
> the right node always?
> Or does the Cassandra nodes redirect queries to the right node after they
> find a right token-range?
> If its the latter, then it seems a bigger cluster size's advantages are
> being offset by an increasing number of redirects between the Cassandra
> nodes.
>
> I tried to look at Cassandra JMX metrics but there are way too many of
> them.
> What metrics should I look into?
>
>
> The tablehistogram for the table being queried looks awesome.
>
> ​
> How do I diagnose this system for a faulty setting?
> What are the usual suspects in such cases?
>
> Thanks
> SG
>
>
>
>
> On Thu, Mar 16, 2017 at 8:11 AM, Patrick McFadin <pm...@gmail.com>
> wrote:
>
>> My general rule of thumb when it comes to stress clients to cluster under
>> load is around 2:1. So with a 12 node cluster, 24 stress clients. As Eric
>> stated, monitoring your stress clients is critical to making sure you have
>> adequate load being produced.
>>
>> Patrick
>>
>> On Wed, Mar 15, 2017 at 3:50 PM, Carlos Rolo <ro...@pythian.com> wrote:
>>
>>> Plus is one of the most documented questions available...
>>>
>>> Regards,
>>>
>>> Carlos Juzarte Rolo
>>> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>>>
>>> Pythian - Love your data
>>>
>>> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
>>> *linkedin.com/in/carlosjuzarterolo
>>> <http://linkedin.com/in/carlosjuzarterolo>*
>>> Mobile: +351 918 918 100 <+351%20918%20918%20100>
>>> www.pythian.com
>>>
>>> On Wed, Mar 15, 2017 at 10:29 PM, James Carman <
>>> james@carmanconsulting.com> wrote:
>>>
>>>> Come on, man.  That's not called for.  Someone is coming here to the
>>>> community for help.  Let's be courteous, please.
>>>>
>>>> On Wed, Mar 15, 2017 at 5:12 PM daemeon reiydelle <da...@gmail.com>
>>>> wrote:
>>>>
>>>>> with that brain damaged memory setting yes.
>>>>>
>>>>>
>>>>> *.......*
>>>>>
>>>>>
>>>>>
>>>>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 <(415)%20501-0198>London
>>>>> (+44) (0) 20 8144 9872 <+44%2020%208144%209872>*
>>>>>
>>>>> On Tue, Mar 14, 2017 at 3:45 PM, S G <sg...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am trying to scale my read throughput.
>>>>>
>>>>> I have a 12 node Cassandra cluster, each node has:
>>>>> RAM: 5.5gb
>>>>> Disk: 64gb
>>>>> C* version: 3.3
>>>>> Java: 1.8.0_51
>>>>>
>>>>> The cluster has 2 tables - each with 2 million rows.
>>>>> Partition keys for all these rows are unique.
>>>>>
>>>>> I am stress-reading it from a 15-node client cluster.
>>>>> Each read-client has 40 threads, so total 600 read-threads from 12
>>>>> machines.
>>>>> Each read query is get-by-primary-key where keys are read randomly
>>>>> from a file having all primary-keys.
>>>>>
>>>>>
>>>>> I am able to get only 15,000 reads/second from the entire system.
>>>>> Is this the best read performance I can expect?
>>>>> Are there any benchmarks I can compare against?
>>>>>
>>>>>
>>>>> I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching
>>>>> to {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
>>>>>
>>>>> I have also captured few JMX metric graphs as follows:
>>>>> [image: Cassandra-Bulk-Read-Metrics.png]
>>>>> ​
>>>>> Garbage collection metrics did not look very interesting, hence I have
>>>>> not added the same.
>>>>>
>>>>> Any ideas what more I can do to debug the system and increase its read
>>>>> throughput?
>>>>>
>>>>> Thanks
>>>>> SG
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Mar 11, 2017 at 2:33 PM, Jeff Jirsa <jj...@gmail.com> wrote:
>>>>>
>>>>> 5.5G of ram isn't very much for a jvm based database. Having fewer
>>>>> instances with more ram will likely give you better performance.
>>>>>
>>>>> Also, this is probably a better question for the user list than the
>>>>> Dev list
>>>>>
>>>>> --
>>>>> Jeff Jirsa
>>>>>
>>>>>
>>>>> > On Mar 11, 2017, at 1:20 PM, S G <sg...@gmail.com> wrote:
>>>>> >
>>>>> > Hi,
>>>>> >
>>>>> > I have a 9 node cassandra cluster, each node has:
>>>>> > RAM: 5.5gb
>>>>> > Disk: 64gb
>>>>> > C* version: 3.3
>>>>> > Java: 1.8.0_51
>>>>> >
>>>>> > The cluster stores about 2 million rows and partition keys for all
>>>>> these
>>>>> > rows are unique.
>>>>> >
>>>>> > I am stress-reading it from a 12-node client cluster.
>>>>> > Each read-client has 50 threads, so total 600 read-threads from 12
>>>>> machines.
>>>>> > Each read query is get-by-primary-key where keys are read randomly
>>>>> from a
>>>>> > file having all primary-keys.
>>>>> >
>>>>> >
>>>>> > I am able to get only 15,000 reads/second from the entire system.
>>>>> > Is this the best read performance I can expect?
>>>>> > Are there any benchmarks I can compare against?
>>>>> >
>>>>> >
>>>>> > I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and
>>>>> caching to
>>>>> > {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
>>>>> >
>>>>> > Thx,
>>>>> > SG
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>> --
>>>
>>>
>>>
>>>
>>
>

Re: How can I scale my read rate?

[Eric Stevens <mi...@gmail.com>]

> I had 12 machines a while ago and when I bumped the cluster size to 16
nodes (33% bigger), there was zero improvement on the read performance.

This is a pretty sure sign that the limiter is your stress testing tool,
and not the cluster.  What was cluster system load like under stress?  You
really should expect it to be well north of 30 or 40.

On Fri, Mar 17, 2017 at 10:35 AM S G <sg...@gmail.com> wrote:

> Thank you all for the nice pointers.
> I am using 22 machines each running 60 threads as reading clients.
> Each client has 8 cores and 23 GB RAM.
>
> On the Cassandra side, I bumped up the machines from 2 core, 5.5gb nodes
> to 8 core, 24 gb nodes and the performance improved from 15,000 to 35,000
> reads/second.
>
> Now it is stuck again at this throughput.
>
> I have tried using G1 garbage collector and played with
> increasing/decreasing threads and machines on the clients reading Cassandra.
> No effect.
>
> Strangest part is this:
> It seems adding more nodes to Cassandra cluster has no effect !
> I had 12 machines a while ago and when I bumped the cluster size to 16
> nodes (33% bigger), there was zero improvement on the read performance.
> Is the datastax client smart enough to route a get-by-primary-key query to
> the right node always?
> Or does the Cassandra nodes redirect queries to the right node after they
> find a right token-range?
> If its the latter, then it seems a bigger cluster size's advantages are
> being offset by an increasing number of redirects between the Cassandra
> nodes.
>
> I tried to look at Cassandra JMX metrics but there are way too many of
> them.
> What metrics should I look into?
>
>
> The tablehistogram for the table being queried looks awesome.
>
> ​
> How do I diagnose this system for a faulty setting?
> What are the usual suspects in such cases?
>
> Thanks
> SG
>
>
>
>
> On Thu, Mar 16, 2017 at 8:11 AM, Patrick McFadin <pm...@gmail.com>
> wrote:
>
> My general rule of thumb when it comes to stress clients to cluster under
> load is around 2:1. So with a 12 node cluster, 24 stress clients. As Eric
> stated, monitoring your stress clients is critical to making sure you have
> adequate load being produced.
>
> Patrick
>
> On Wed, Mar 15, 2017 at 3:50 PM, Carlos Rolo <ro...@pythian.com> wrote:
>
> Plus is one of the most documented questions available...
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
> *linkedin.com/in/carlosjuzarterolo
> <http://linkedin.com/in/carlosjuzarterolo>*
> Mobile: +351 918 918 100 <+351%20918%20918%20100>
> www.pythian.com
>
> On Wed, Mar 15, 2017 at 10:29 PM, James Carman <james@carmanconsulting.com
> > wrote:
>
> Come on, man.  That's not called for.  Someone is coming here to the
> community for help.  Let's be courteous, please.
>
> On Wed, Mar 15, 2017 at 5:12 PM daemeon reiydelle <da...@gmail.com>
> wrote:
>
> with that brain damaged memory setting yes.
>
>
> *.......*
>
>
>
> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 <(415)%20501-0198>London
> (+44) (0) 20 8144 9872 <+44%2020%208144%209872>*
>
> On Tue, Mar 14, 2017 at 3:45 PM, S G <sg...@gmail.com> wrote:
>
> Hi,
>
> I am trying to scale my read throughput.
>
> I have a 12 node Cassandra cluster, each node has:
> RAM: 5.5gb
> Disk: 64gb
> C* version: 3.3
> Java: 1.8.0_51
>
> The cluster has 2 tables - each with 2 million rows.
> Partition keys for all these rows are unique.
>
> I am stress-reading it from a 15-node client cluster.
> Each read-client has 40 threads, so total 600 read-threads from 12
> machines.
> Each read query is get-by-primary-key where keys are read randomly from a
> file having all primary-keys.
>
>
> I am able to get only 15,000 reads/second from the entire system.
> Is this the best read performance I can expect?
> Are there any benchmarks I can compare against?
>
>
> I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching to
> {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
>
> I have also captured few JMX metric graphs as follows:
> [image: Cassandra-Bulk-Read-Metrics.png]
> ​
> Garbage collection metrics did not look very interesting, hence I have not
> added the same.
>
> Any ideas what more I can do to debug the system and increase its read
> throughput?
>
> Thanks
> SG
>
>
>
> On Sat, Mar 11, 2017 at 2:33 PM, Jeff Jirsa <jj...@gmail.com> wrote:
>
> 5.5G of ram isn't very much for a jvm based database. Having fewer
> instances with more ram will likely give you better performance.
>
> Also, this is probably a better question for the user list than the Dev
> list
>
> --
> Jeff Jirsa
>
>
> > On Mar 11, 2017, at 1:20 PM, S G <sg...@gmail.com> wrote:
> >
> > Hi,
> >
> > I have a 9 node cassandra cluster, each node has:
> > RAM: 5.5gb
> > Disk: 64gb
> > C* version: 3.3
> > Java: 1.8.0_51
> >
> > The cluster stores about 2 million rows and partition keys for all these
> > rows are unique.
> >
> > I am stress-reading it from a 12-node client cluster.
> > Each read-client has 50 threads, so total 600 read-threads from 12
> machines.
> > Each read query is get-by-primary-key where keys are read randomly from a
> > file having all primary-keys.
> >
> >
> > I am able to get only 15,000 reads/second from the entire system.
> > Is this the best read performance I can expect?
> > Are there any benchmarks I can compare against?
> >
> >
> > I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching to
> > {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
> >
> > Thx,
> > SG
>
>
>
>
>
> --
>
>
>
>
>
>

Re: How can I scale my read rate?

[S G <sg...@gmail.com>]

Thank you all for the nice pointers.
I am using 22 machines each running 60 threads as reading clients.
Each client has 8 cores and 23 GB RAM.

On the Cassandra side, I bumped up the machines from 2 core, 5.5gb nodes to
8 core, 24 gb nodes and the performance improved from 15,000 to 35,000
reads/second.

Now it is stuck again at this throughput.

I have tried using G1 garbage collector and played with
increasing/decreasing threads and machines on the clients reading Cassandra.
No effect.

Strangest part is this:
It seems adding more nodes to Cassandra cluster has no effect !
I had 12 machines a while ago and when I bumped the cluster size to 16
nodes (33% bigger), there was zero improvement on the read performance.
Is the datastax client smart enough to route a get-by-primary-key query to
the right node always?
Or does the Cassandra nodes redirect queries to the right node after they
find a right token-range?
If its the latter, then it seems a bigger cluster size's advantages are
being offset by an increasing number of redirects between the Cassandra
nodes.

I tried to look at Cassandra JMX metrics but there are way too many of them.
What metrics should I look into?


The tablehistogram for the table being queried looks awesome.

​
How do I diagnose this system for a faulty setting?
What are the usual suspects in such cases?

Thanks
SG




On Thu, Mar 16, 2017 at 8:11 AM, Patrick McFadin <pm...@gmail.com> wrote:

> My general rule of thumb when it comes to stress clients to cluster under
> load is around 2:1. So with a 12 node cluster, 24 stress clients. As Eric
> stated, monitoring your stress clients is critical to making sure you have
> adequate load being produced.
>
> Patrick
>
> On Wed, Mar 15, 2017 at 3:50 PM, Carlos Rolo <ro...@pythian.com> wrote:
>
>> Plus is one of the most documented questions available...
>>
>> Regards,
>>
>> Carlos Juzarte Rolo
>> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>>
>> Pythian - Love your data
>>
>> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
>> *linkedin.com/in/carlosjuzarterolo
>> <http://linkedin.com/in/carlosjuzarterolo>*
>> Mobile: +351 918 918 100 <+351%20918%20918%20100>
>> www.pythian.com
>>
>> On Wed, Mar 15, 2017 at 10:29 PM, James Carman <
>> james@carmanconsulting.com> wrote:
>>
>>> Come on, man.  That's not called for.  Someone is coming here to the
>>> community for help.  Let's be courteous, please.
>>>
>>> On Wed, Mar 15, 2017 at 5:12 PM daemeon reiydelle <da...@gmail.com>
>>> wrote:
>>>
>>>> with that brain damaged memory setting yes.
>>>>
>>>>
>>>> *.......*
>>>>
>>>>
>>>>
>>>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 <(415)%20501-0198>London
>>>> (+44) (0) 20 8144 9872 <+44%2020%208144%209872>*
>>>>
>>>> On Tue, Mar 14, 2017 at 3:45 PM, S G <sg...@gmail.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I am trying to scale my read throughput.
>>>>
>>>> I have a 12 node Cassandra cluster, each node has:
>>>> RAM: 5.5gb
>>>> Disk: 64gb
>>>> C* version: 3.3
>>>> Java: 1.8.0_51
>>>>
>>>> The cluster has 2 tables - each with 2 million rows.
>>>> Partition keys for all these rows are unique.
>>>>
>>>> I am stress-reading it from a 15-node client cluster.
>>>> Each read-client has 40 threads, so total 600 read-threads from 12
>>>> machines.
>>>> Each read query is get-by-primary-key where keys are read randomly from
>>>> a file having all primary-keys.
>>>>
>>>>
>>>> I am able to get only 15,000 reads/second from the entire system.
>>>> Is this the best read performance I can expect?
>>>> Are there any benchmarks I can compare against?
>>>>
>>>>
>>>> I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching
>>>> to {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
>>>>
>>>> I have also captured few JMX metric graphs as follows:
>>>> [image: Cassandra-Bulk-Read-Metrics.png]
>>>> ​
>>>> Garbage collection metrics did not look very interesting, hence I have
>>>> not added the same.
>>>>
>>>> Any ideas what more I can do to debug the system and increase its read
>>>> throughput?
>>>>
>>>> Thanks
>>>> SG
>>>>
>>>>
>>>>
>>>> On Sat, Mar 11, 2017 at 2:33 PM, Jeff Jirsa <jj...@gmail.com> wrote:
>>>>
>>>> 5.5G of ram isn't very much for a jvm based database. Having fewer
>>>> instances with more ram will likely give you better performance.
>>>>
>>>> Also, this is probably a better question for the user list than the Dev
>>>> list
>>>>
>>>> --
>>>> Jeff Jirsa
>>>>
>>>>
>>>> > On Mar 11, 2017, at 1:20 PM, S G <sg...@gmail.com> wrote:
>>>> >
>>>> > Hi,
>>>> >
>>>> > I have a 9 node cassandra cluster, each node has:
>>>> > RAM: 5.5gb
>>>> > Disk: 64gb
>>>> > C* version: 3.3
>>>> > Java: 1.8.0_51
>>>> >
>>>> > The cluster stores about 2 million rows and partition keys for all
>>>> these
>>>> > rows are unique.
>>>> >
>>>> > I am stress-reading it from a 12-node client cluster.
>>>> > Each read-client has 50 threads, so total 600 read-threads from 12
>>>> machines.
>>>> > Each read query is get-by-primary-key where keys are read randomly
>>>> from a
>>>> > file having all primary-keys.
>>>> >
>>>> >
>>>> > I am able to get only 15,000 reads/second from the entire system.
>>>> > Is this the best read performance I can expect?
>>>> > Are there any benchmarks I can compare against?
>>>> >
>>>> >
>>>> > I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and
>>>> caching to
>>>> > {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
>>>> >
>>>> > Thx,
>>>> > SG
>>>>
>>>>
>>>>
>>>>
>>
>> --
>>
>>
>>
>>
>

Re: How can I scale my read rate?

[Patrick McFadin <pm...@gmail.com>]

My general rule of thumb when it comes to stress clients to cluster under
load is around 2:1. So with a 12 node cluster, 24 stress clients. As Eric
stated, monitoring your stress clients is critical to making sure you have
adequate load being produced.

Patrick

On Wed, Mar 15, 2017 at 3:50 PM, Carlos Rolo <ro...@pythian.com> wrote:

> Plus is one of the most documented questions available...
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
> *linkedin.com/in/carlosjuzarterolo
> <http://linkedin.com/in/carlosjuzarterolo>*
> Mobile: +351 918 918 100 <+351%20918%20918%20100>
> www.pythian.com
>
> On Wed, Mar 15, 2017 at 10:29 PM, James Carman <james@carmanconsulting.com
> > wrote:
>
>> Come on, man.  That's not called for.  Someone is coming here to the
>> community for help.  Let's be courteous, please.
>>
>> On Wed, Mar 15, 2017 at 5:12 PM daemeon reiydelle <da...@gmail.com>
>> wrote:
>>
>>> with that brain damaged memory setting yes.
>>>
>>>
>>> *.......*
>>>
>>>
>>>
>>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 <(415)%20501-0198>London
>>> (+44) (0) 20 8144 9872 <+44%2020%208144%209872>*
>>>
>>> On Tue, Mar 14, 2017 at 3:45 PM, S G <sg...@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> I am trying to scale my read throughput.
>>>
>>> I have a 12 node Cassandra cluster, each node has:
>>> RAM: 5.5gb
>>> Disk: 64gb
>>> C* version: 3.3
>>> Java: 1.8.0_51
>>>
>>> The cluster has 2 tables - each with 2 million rows.
>>> Partition keys for all these rows are unique.
>>>
>>> I am stress-reading it from a 15-node client cluster.
>>> Each read-client has 40 threads, so total 600 read-threads from 12
>>> machines.
>>> Each read query is get-by-primary-key where keys are read randomly from
>>> a file having all primary-keys.
>>>
>>>
>>> I am able to get only 15,000 reads/second from the entire system.
>>> Is this the best read performance I can expect?
>>> Are there any benchmarks I can compare against?
>>>
>>>
>>> I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching
>>> to {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
>>>
>>> I have also captured few JMX metric graphs as follows:
>>> [image: Cassandra-Bulk-Read-Metrics.png]
>>> ​
>>> Garbage collection metrics did not look very interesting, hence I have
>>> not added the same.
>>>
>>> Any ideas what more I can do to debug the system and increase its read
>>> throughput?
>>>
>>> Thanks
>>> SG
>>>
>>>
>>>
>>> On Sat, Mar 11, 2017 at 2:33 PM, Jeff Jirsa <jj...@gmail.com> wrote:
>>>
>>> 5.5G of ram isn't very much for a jvm based database. Having fewer
>>> instances with more ram will likely give you better performance.
>>>
>>> Also, this is probably a better question for the user list than the Dev
>>> list
>>>
>>> --
>>> Jeff Jirsa
>>>
>>>
>>> > On Mar 11, 2017, at 1:20 PM, S G <sg...@gmail.com> wrote:
>>> >
>>> > Hi,
>>> >
>>> > I have a 9 node cassandra cluster, each node has:
>>> > RAM: 5.5gb
>>> > Disk: 64gb
>>> > C* version: 3.3
>>> > Java: 1.8.0_51
>>> >
>>> > The cluster stores about 2 million rows and partition keys for all
>>> these
>>> > rows are unique.
>>> >
>>> > I am stress-reading it from a 12-node client cluster.
>>> > Each read-client has 50 threads, so total 600 read-threads from 12
>>> machines.
>>> > Each read query is get-by-primary-key where keys are read randomly
>>> from a
>>> > file having all primary-keys.
>>> >
>>> >
>>> > I am able to get only 15,000 reads/second from the entire system.
>>> > Is this the best read performance I can expect?
>>> > Are there any benchmarks I can compare against?
>>> >
>>> >
>>> > I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching
>>> to
>>> > {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
>>> >
>>> > Thx,
>>> > SG
>>>
>>>
>>>
>>>
>
> --
>
>
>
>

Re: How can I scale my read rate?

[Carlos Rolo <ro...@pythian.com>]

Plus is one of the most documented questions available...

Regards,

Carlos Juzarte Rolo
Cassandra Consultant / Datastax Certified Architect / Cassandra MVP

Pythian - Love your data

rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
*linkedin.com/in/carlosjuzarterolo
<http://linkedin.com/in/carlosjuzarterolo>*
Mobile: +351 918 918 100
www.pythian.com

On Wed, Mar 15, 2017 at 10:29 PM, James Carman <ja...@carmanconsulting.com>
wrote:

> Come on, man.  That's not called for.  Someone is coming here to the
> community for help.  Let's be courteous, please.
>
> On Wed, Mar 15, 2017 at 5:12 PM daemeon reiydelle <da...@gmail.com>
> wrote:
>
>> with that brain damaged memory setting yes.
>>
>>
>> *.......*
>>
>>
>>
>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 <(415)%20501-0198>London
>> (+44) (0) 20 8144 9872 <+44%2020%208144%209872>*
>>
>> On Tue, Mar 14, 2017 at 3:45 PM, S G <sg...@gmail.com> wrote:
>>
>> Hi,
>>
>> I am trying to scale my read throughput.
>>
>> I have a 12 node Cassandra cluster, each node has:
>> RAM: 5.5gb
>> Disk: 64gb
>> C* version: 3.3
>> Java: 1.8.0_51
>>
>> The cluster has 2 tables - each with 2 million rows.
>> Partition keys for all these rows are unique.
>>
>> I am stress-reading it from a 15-node client cluster.
>> Each read-client has 40 threads, so total 600 read-threads from 12
>> machines.
>> Each read query is get-by-primary-key where keys are read randomly from a
>> file having all primary-keys.
>>
>>
>> I am able to get only 15,000 reads/second from the entire system.
>> Is this the best read performance I can expect?
>> Are there any benchmarks I can compare against?
>>
>>
>> I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching to
>> {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
>>
>> I have also captured few JMX metric graphs as follows:
>> [image: Cassandra-Bulk-Read-Metrics.png]
>> ​
>> Garbage collection metrics did not look very interesting, hence I have
>> not added the same.
>>
>> Any ideas what more I can do to debug the system and increase its read
>> throughput?
>>
>> Thanks
>> SG
>>
>>
>>
>> On Sat, Mar 11, 2017 at 2:33 PM, Jeff Jirsa <jj...@gmail.com> wrote:
>>
>> 5.5G of ram isn't very much for a jvm based database. Having fewer
>> instances with more ram will likely give you better performance.
>>
>> Also, this is probably a better question for the user list than the Dev
>> list
>>
>> --
>> Jeff Jirsa
>>
>>
>> > On Mar 11, 2017, at 1:20 PM, S G <sg...@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > I have a 9 node cassandra cluster, each node has:
>> > RAM: 5.5gb
>> > Disk: 64gb
>> > C* version: 3.3
>> > Java: 1.8.0_51
>> >
>> > The cluster stores about 2 million rows and partition keys for all these
>> > rows are unique.
>> >
>> > I am stress-reading it from a 12-node client cluster.
>> > Each read-client has 50 threads, so total 600 read-threads from 12
>> machines.
>> > Each read query is get-by-primary-key where keys are read randomly from
>> a
>> > file having all primary-keys.
>> >
>> >
>> > I am able to get only 15,000 reads/second from the entire system.
>> > Is this the best read performance I can expect?
>> > Are there any benchmarks I can compare against?
>> >
>> >
>> > I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching
>> to
>> > {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
>> >
>> > Thx,
>> > SG
>>
>>
>>
>>

-- 


--

Re: How can I scale my read rate?

[James Carman <ja...@carmanconsulting.com>]

Come on, man.  That's not called for.  Someone is coming here to the
community for help.  Let's be courteous, please.

On Wed, Mar 15, 2017 at 5:12 PM daemeon reiydelle <da...@gmail.com>
wrote:

> with that brain damaged memory setting yes.
>
>
> *.......*
>
>
>
> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 <(415)%20501-0198>London
> (+44) (0) 20 8144 9872 <+44%2020%208144%209872>*
>
> On Tue, Mar 14, 2017 at 3:45 PM, S G <sg...@gmail.com> wrote:
>
> Hi,
>
> I am trying to scale my read throughput.
>
> I have a 12 node Cassandra cluster, each node has:
> RAM: 5.5gb
> Disk: 64gb
> C* version: 3.3
> Java: 1.8.0_51
>
> The cluster has 2 tables - each with 2 million rows.
> Partition keys for all these rows are unique.
>
> I am stress-reading it from a 15-node client cluster.
> Each read-client has 40 threads, so total 600 read-threads from 12
> machines.
> Each read query is get-by-primary-key where keys are read randomly from a
> file having all primary-keys.
>
>
> I am able to get only 15,000 reads/second from the entire system.
> Is this the best read performance I can expect?
> Are there any benchmarks I can compare against?
>
>
> I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching to
> {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
>
> I have also captured few JMX metric graphs as follows:
> [image: Cassandra-Bulk-Read-Metrics.png]
> ​
> Garbage collection metrics did not look very interesting, hence I have not
> added the same.
>
> Any ideas what more I can do to debug the system and increase its read
> throughput?
>
> Thanks
> SG
>
>
>
> On Sat, Mar 11, 2017 at 2:33 PM, Jeff Jirsa <jj...@gmail.com> wrote:
>
> 5.5G of ram isn't very much for a jvm based database. Having fewer
> instances with more ram will likely give you better performance.
>
> Also, this is probably a better question for the user list than the Dev
> list
>
> --
> Jeff Jirsa
>
>
> > On Mar 11, 2017, at 1:20 PM, S G <sg...@gmail.com> wrote:
> >
> > Hi,
> >
> > I have a 9 node cassandra cluster, each node has:
> > RAM: 5.5gb
> > Disk: 64gb
> > C* version: 3.3
> > Java: 1.8.0_51
> >
> > The cluster stores about 2 million rows and partition keys for all these
> > rows are unique.
> >
> > I am stress-reading it from a 12-node client cluster.
> > Each read-client has 50 threads, so total 600 read-threads from 12
> machines.
> > Each read query is get-by-primary-key where keys are read randomly from a
> > file having all primary-keys.
> >
> >
> > I am able to get only 15,000 reads/second from the entire system.
> > Is this the best read performance I can expect?
> > Are there any benchmarks I can compare against?
> >
> >
> > I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching to
> > {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
> >
> > Thx,
> > SG
>
>
>
>

AW: How can I scale my read rate?

[j....@enercast.de]

+1 for executeAsync – had a long time to argue that it’s not bad as with good old rdbms. 



Gesendet von meinem Windows 10 Phone

Von: Arvydas Jonusonis
Gesendet: Samstag, 18. März 2017 19:08
An: user@cassandra.apache.org
Betreff: Re: How can I scale my read rate?

..then you're not taking advantage of request pipelining. Use executeAsync - this will increase your throughput for sure.

http://www.datastax.com/dev/blog/java-driver-async-queries


On Sat, Mar 18, 2017 at 08:00 S G <sg...@gmail.com> wrote:
I have enabled JMX but not sure what metrics to look for - they are way too many of them.
I am using session.execute(...)


On Fri, Mar 17, 2017 at 2:07 PM, Arvydas Jonusonis <ar...@gmail.com> wrote:
It would be interesting to see some of the driver metrics (in your stress test tool) - if you enable JMX, they should be exposed by default.

Also, are you using session.execute(..) or session.executeAsync(..) ?

Re: How can I scale my read rate?

[Alain Rastoul <al...@gmail.com>]

On 19/03/2017 02:54, S G wrote:
> Forgot to mention that this vmstat picture is for the client-cluster
> reading from Cassandra.
>

Hi SG,


Your numbers are low, 15k req/sec would be ok for a single node, for a 
12 nodes cluster, something goes wrong... how do you measure the 
throughput?

As suggested by others, to achieve good results you have to add threads 
and client VMs: Cassandra scales horizontally, not vertically, ie each 
single node performance will not goes up, but if you spread the load, by 
adding nodes the global cluster performance will.

Theorically,
assuming the data and the load is spread on the cluster (*1)
from your saying, with each request at 2ms avg (*2)
you should have 500 req/sec in each thread,
40 threads should go 20k req/sec on each client VM stress application (*3)
and 10 client VMs should go 200k req/sec on the whole cluster (*4)

=====
(*1) the partition key (first PK column) must spread data on all nodes
and your testing code must spread the load by selecting evenly spread data.
(This point is very important: can you give information on your schema 
and your data ?)

(*2) to achieve better single client throughtput, may be you could 
prepare the requests, since you are always executing the same requests

(*3) => run more client tests application on each VM

(*4) add more client  VMs (Patrick's suggestion)

with (3) and (4) the throughput of each client will not be better, but 
the global cluster throughput will.
=====

There are other factors to take into account if you are also writing to 
the cluster : read path, tombstones, replication, repairs etc. but 
that's not the case here?

Performance testing goes to the limit of our understanding of the system
  and is very difficult
... hence interesting :)



-- 
best,
Alain

Re: How can I scale my read rate?

[S G <sg...@gmail.com>]

Forgot to mention that this vmstat picture is for the client-cluster
reading from Cassandra.

On Sat, Mar 18, 2017 at 6:47 PM, S G <sg...@gmail.com> wrote:

> ok, I gave the executeAsync() a try.
> Good part is that it was really easy to write the code for that.
> Bad part is that it did not had a huge effect on my throughput - I gained
> about 5% increase in throughput.
> I suspect it is so because my queries are all get-by-primary-key queries
> and were anyways completing in less than 2 milliseconds.
> So there was not much wait to begin with.
>
>
> Here is my code:
>
> String getByKeyQueryStr = "Select * from fooTable where key = " + key;
> //ResultSet result = session.execute(getByKeyQueryStr);  // Previous code
> ResultSetFuture future = session.executeAsync(getByKeyQueryStr);
> FutureCallback<ResultSet> callback = new MyFutureCallback();
> executor = MoreExecutors.sameThreadExecutor();
> //executor = Executors.newFixedThreadPool(3); // Tried this too, no effect
> //executor = Executors.newFixedThreadPool(10); // Tried this too, no
> effect
> Futures.addCallback(future, callback, executor);
>
> Can I improve the above code in some way?
> Are there any JMX metrics that can tell me what's going on?
>
> From the vmstat command, I see that CPU idle time is about 70% even though
> I am running about 60 threads per VM
> Total 20 client-VMs with 8 cores each are querying a Cassandra cluster
> with 16 VMs, 8-core each too.
>
>
> ​
> ​
>
>
> Thanks
> SG
>
>
> On Sat, Mar 18, 2017 at 5:38 PM, S G <sg...@gmail.com> wrote:
>
>> Thanks. It seems that you guys have found executeAsync to yield good
>> results.
>> I want to share my understanding how this could benefit performance and
>> some validation from the group will be awesome.
>>
>> I will call executeAsync() each time I want to get by primary-key.
>> That ways, my client thread is not blocked anymore and I can submit a lot
>> more requests per unit time.
>> The async requests get piled on the underlying Netty I/O thread which
>> ensures that it is always busy all the time.
>> Earlier, the Netty I/O thread would have wasted some cycles when the
>> sync-execute method was processing the results.
>> And earlier, the client thread would also have wasted some cycles waiting
>> for netty-thread to complete.
>>
>> With executeAsync(), none of them is waiting.
>> Only thing to ensure is that the Netty thread's queue does not grow
>> indefinitely.
>>
>> If the above theory is correct, then it sounds like a really good thing
>> to try.
>> If not, please do share some more details.
>>
>>
>>
>>
>> On Sat, Mar 18, 2017 at 2:00 PM, <j....@enercast.de> wrote:
>>
>>> +1 for executeAsync – had a long time to argue that it’s not bad as with
>>> good old rdbms.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Gesendet von meinem Windows 10 Phone
>>>
>>>
>>>
>>> *Von: *Arvydas Jonusonis <ar...@gmail.com>
>>> *Gesendet: *Samstag, 18. März 2017 19:08
>>> *An: *user@cassandra.apache.org
>>> *Betreff: *Re: How can I scale my read rate?
>>>
>>>
>>>
>>> ..then you're not taking advantage of request pipelining. Use
>>> executeAsync - this will increase your throughput for sure.
>>>
>>>
>>>
>>> http://www.datastax.com/dev/blog/java-driver-async-queries
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Mar 18, 2017 at 08:00 S G <sg...@gmail.com> wrote:
>>>
>>> I have enabled JMX but not sure what metrics to look for - they are way
>>> too many of them.
>>>
>>> I am using session.execute(...)
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Mar 17, 2017 at 2:07 PM, Arvydas Jonusonis <
>>> arvydas.jonusonis@gmail.com> wrote:
>>>
>>> It would be interesting to see some of the driver metrics (in your
>>> stress test tool) - if you enable JMX, they should be exposed by default.
>>>
>>> Also, are you using session.execute(..) or session.executeAsync(..) ?
>>>
>>>
>>>
>>>
>>>
>>
>

Re: How can I scale my read rate?

[Alain Rastoul <al...@gmail.com>]

On 20/03/2017 02:35, S G wrote:
> 2)
> https://docs.datastax.com/en/developer/java-driver/3.1/manual/statements/prepared/
> tells me to avoid preparing select queries if I expect a change of
> columns in my table down the road.
The problem is also related to select * which is considered bad practice 
with most databases...

> I did some more testing to see if my client machines were the bottleneck.
> For a 6-node Cassandra cluster (each VM having 8-cores), I got 26,000
> reads/sec for all of the following:
> 1) Client nodes:1, Threads: 60
> 2) Client nodes:3, Threads: 180
> 3) Client nodes:5, Threads: 300
> 4) Client nodes:10, Threads: 600
> 5) Client nodes:20, Threads: 1200
>
> So adding more client nodes or threads to those client nodes is not
> having any effect.
> I am suspecting Cassandra is simply not allowing me to go any further.
 > Primary keys for my schema are:
 >      PRIMARY KEY((name, phone), age)
 > name: text
 > phone: int
 > age: int

Yes with such a PK data must be spread on the whole cluster (also taking 
into account the partitioner), strange that the throughput doesn't scale.
I guess you also have verified that you select data randomly?

May be you could have a look at the system traces to see the query plan 
for some requests:
If you are on a test cluster you can truncate the tables before 
(truncate system_traces.sessions; and truncate system_traces.events;), 
run a test then select * from system_traces.events
where session_id = xxxx
xxx being one of the sessions you pick in trace.sessions.

Try to see if you are not always hitting the same nodes.


-- 
best,
Alain

Re: How can I scale my read rate?

[James Carman <ja...@carmanconsulting.com>]

It says to avoid "select *" if you anticipate the columns will change.  I
never use "select*".  How tough would it be to just try the prepared
statement in your code?  Even a 20% increase would help you push more load
through your cluster, right?  I'd try it.  I tend to *always* use prepared
statements, personally.  Perhaps that's just me, but I even find them
easier to deal with.
On Sun, Mar 19, 2017 at 9:43 PM S G <sg...@gmail.com> wrote:

> I am not using prepared statements. There are two reasons for that:
>
> 1) https://issues.apache.org/jira/browse/CASSANDRA-3634 tells me that the
> performance improvements with prepared statements are capped at about 20% -
> so I would not see a drastic difference.
> 2)
> https://docs.datastax.com/en/developer/java-driver/3.1/manual/statements/prepared/
> tells me to avoid preparing select queries if I expect a change of columns
> in my table down the road.
>
> So I wasn't very keen on trying those out.
> Please suggest if I can expect to get more than 20% performance using the
> same and I will try that too.
>
>
> I did some more testing to see if my client machines were the bottleneck.
> For a 6-node Cassandra cluster (each VM having 8-cores), I got 26,000
> reads/sec for all of the following:
> 1) Client nodes:1, Threads: 60
> 2) Client nodes:3, Threads: 180
> 3) Client nodes:5, Threads: 300
> 4) Client nodes:10, Threads: 600
> 5) Client nodes:20, Threads: 1200
>
> So adding more client nodes or threads to those client nodes is not having
> any effect.
> I am suspecting Cassandra is simply not allowing me to go any further.
>
> Reads/second is measured both by the opscenter and by my own logic in the
> client machines code. They seem to agree more or less.
>
> Primary keys for my schema are:
>     PRIMARY KEY((name, phone), age)
> name: text
> phone: int
> age: int
>
> And there are very few columns in the table apart from the above. None of
> them is more than few bytes long.
>
> 1) Does Cassandra have something like debug=timing
> <http://stackoverflow.com/questions/14712175/splitting-solr-response-time>
> in Solr which gives all the information about where query-time was spent?
> 2) Does the nodetool have something that can give me some hint?
> 3) Is there a JMX metrics I should look for?
>
>
> Thanks
> SG
>
>
>
>
>
> On Sun, Mar 19, 2017 at 5:03 AM, James Carman <ja...@carmanconsulting.com>
> wrote:
>
> Have you tried using PreparedStatements?
>
> On Sat, Mar 18, 2017 at 9:47 PM S G <sg...@gmail.com> wrote:
>
> ok, I gave the executeAsync() a try.
> Good part is that it was really easy to write the code for that.
> Bad part is that it did not had a huge effect on my throughput - I gained
> about 5% increase in throughput.
> I suspect it is so because my queries are all get-by-primary-key queries
> and were anyways completing in less than 2 milliseconds.
> So there was not much wait to begin with.
>
>
> Here is my code:
>
> String getByKeyQueryStr = "Select * from fooTable where key = " + key;
> //ResultSet result = session.execute(getByKeyQueryStr);  // Previous code
> ResultSetFuture future = session.executeAsync(getByKeyQueryStr);
> FutureCallback<ResultSet> callback = new MyFutureCallback();
> executor = MoreExecutors.sameThreadExecutor();
> //executor = Executors.newFixedThreadPool(3); // Tried this too, no effect
> //executor = Executors.newFixedThreadPool(10); // Tried this too, no effect
> Futures.addCallback(future, callback, executor);
>
> Can I improve the above code in some way?
> Are there any JMX metrics that can tell me what's going on?
>
> From the vmstat command, I see that CPU idle time is about 70% even though
> I am running about 60 threads per VM
> Total 20 client-VMs with 8 cores each are querying a Cassandra cluster
> with 16 VMs, 8-core each too.
>
> [image: Screen Shot 2017-03-18 at 6.46.03 PM.png]
> ​
> ​
>
>
> Thanks
> SG
>
>
> On Sat, Mar 18, 2017 at 5:38 PM, S G <sg...@gmail.com> wrote:
>
> Thanks. It seems that you guys have found executeAsync to yield good
> results.
> I want to share my understanding how this could benefit performance and
> some validation from the group will be awesome.
>
> I will call executeAsync() each time I want to get by primary-key.
> That ways, my client thread is not blocked anymore and I can submit a lot
> more requests per unit time.
> The async requests get piled on the underlying Netty I/O thread which
> ensures that it is always busy all the time.
> Earlier, the Netty I/O thread would have wasted some cycles when the
> sync-execute method was processing the results.
> And earlier, the client thread would also have wasted some cycles waiting
> for netty-thread to complete.
>
> With executeAsync(), none of them is waiting.
> Only thing to ensure is that the Netty thread's queue does not grow
> indefinitely.
>
> If the above theory is correct, then it sounds like a really good thing to
> try.
> If not, please do share some more details.
>
>
>
>
> On Sat, Mar 18, 2017 at 2:00 PM, <j....@enercast.de> wrote:
>
> +1 for executeAsync – had a long time to argue that it’s not bad as with
> good old rdbms.
>
>
>
>
>
>
>
> Gesendet von meinem Windows 10 Phone
>
>
>
> *Von: *Arvydas Jonusonis <ar...@gmail.com>
> *Gesendet: *Samstag, 18. März 2017 19:08
> *An: *user@cassandra.apache.org
> *Betreff: *Re: How can I scale my read rate?
>
>
>
> ..then you're not taking advantage of request pipelining. Use executeAsync
> - this will increase your throughput for sure.
>
>
>
> http://www.datastax.com/dev/blog/java-driver-async-queries
>
>
>
>
>
> On Sat, Mar 18, 2017 at 08:00 S G <sg...@gmail.com> wrote:
>
> I have enabled JMX but not sure what metrics to look for - they are way
> too many of them.
>
> I am using session.execute(...)
>
>
>
>
>
> On Fri, Mar 17, 2017 at 2:07 PM, Arvydas Jonusonis <
> arvydas.jonusonis@gmail.com> wrote:
>
> It would be interesting to see some of the driver metrics (in your stress
> test tool) - if you enable JMX, they should be exposed by default.
>
> Also, are you using session.execute(..) or session.executeAsync(..) ?
>
>
>
>
>
>
>
>

Re: How can I scale my read rate?

[Alexander Dejanovski <al...@thelastpickle.com>]

By default the TokenAwarePolicy does shuffle replicas, and it can be
disabled if you want to only hit the primary replica for the token range
you're querying :
http://docs.datastax.com/en/drivers/java/3.0/com/datastax/driver/core/policies/TokenAwarePolicy.html

On Mon, Mar 27, 2017 at 9:41 AM Avi Kivity <av...@scylladb.com> wrote:

> Is the driver doing the right thing by directing all reads for a given
> token to the same node?  If that node fails, then all of those reads will
> be directed at other nodes, all oh whom will be cache-cold for the the
> failed node's primary token range.  Seems like the driver should distribute
> reads among the all the replicas for a token, at least as an option, to
> keep the caches warm for latency-sensitive loads.
>
> On 03/26/2017 07:46 PM, Eric Stevens wrote:
>
> Yes, throughput for a given partition key cannot be improved with
> horizontal scaling.  You can increase RF to theoretically improve
> throughput on that key, but actually in this case smart clients might hold
> you back, because they're probably token aware, and will try to serve that
> read off the key's primary replica, so all reads would be directed at a
> single node for that key.
>
> If you're reading at CL=QUORUM, there's a chance that increasing RF will
> actually reduce performance rather than improve it, because you've
> increased the total amount of work to serve the read (as well as the
> write).  If you're reading at CL=ONE, increasing RF will increase the
> chances of falling afoul of eventual consistency.
>
> However that's not really a real-world scenario.  Or if it is, Cassandra
> is probably the wrong tool to satisfy that kind of workload.
>
> On Thu, Mar 23, 2017 at 11:43 PM Alain Rastoul <al...@gmail.com>
> wrote:
>
> On 24/03/2017 01:00, Eric Stevens wrote:
> > Assuming an even distribution of data in your cluster, and an even
> > distribution across those keys by your readers, you would not need to
> > increase RF with cluster size to increase read performance.  If you have
> > 3 nodes with RF=3, and do 3 million reads, with good distribution, each
> > node has served 1 million read requests.  If you increase to 6 nodes and
> > keep RF=3, then each node now owns half as much data and serves only
> > 500,000 reads.  Or more meaningfully in the same time it takes to do 3
> > million reads under the 3 node cluster you ought to be able to do 6
> > million reads under the 6 node cluster since each node is just
> > responsible for 1 million total reads.
> >
> Hi Eric,
>
> I think I got your point.
> In case of really evenly distributed  reads it may (or should?) not make
> any difference,
>
> But when you do not distribute well the reads (and in that case only),
> my understanding about RF was that it could help spreading the load :
> In that case, with RF= 4 instead of 3,  with several clients accessing keys
> same key ranges, a coordinator could pick up one node to handle the request
> in 4 replicas instead of picking up one node in 3 , thus having
> more "workers" to handle a request ?
>
> Am I wrong here ?
>
> Thank you for the clarification
>
>
> --
> best,
> Alain
>
>
> --
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: How can I scale my read rate?

[Avi Kivity <av...@scylladb.com>]

Is the driver doing the right thing by directing all reads for a given 
token to the same node?  If that node fails, then all of those reads 
will be directed at other nodes, all oh whom will be cache-cold for the 
the failed node's primary token range.  Seems like the driver should 
distribute reads among the all the replicas for a token, at least as an 
option, to keep the caches warm for latency-sensitive loads.


On 03/26/2017 07:46 PM, Eric Stevens wrote:
> Yes, throughput for a given partition key cannot be improved with 
> horizontal scaling.  You can increase RF to theoretically improve 
> throughput on that key, but actually in this case smart clients might 
> hold you back, because they're probably token aware, and will try to 
> serve that read off the key's primary replica, so all reads would be 
> directed at a single node for that key.
>
> If you're reading at CL=QUORUM, there's a chance that increasing RF 
> will actually reduce performance rather than improve it, because 
> you've increased the total amount of work to serve the read (as well 
> as the write).  If you're reading at CL=ONE, increasing RF will 
> increase the chances of falling afoul of eventual consistency.
>
> However that's not really a real-world scenario.  Or if it is, 
> Cassandra is probably the wrong tool to satisfy that kind of workload.
>
> On Thu, Mar 23, 2017 at 11:43 PM Alain Rastoul <alf.mmm.cat@gmail.com 
> <ma...@gmail.com>> wrote:
>
>     On 24/03/2017 01:00, Eric Stevens wrote:
>     > Assuming an even distribution of data in your cluster, and an even
>     > distribution across those keys by your readers, you would not
>     need to
>     > increase RF with cluster size to increase read performance.  If
>     you have
>     > 3 nodes with RF=3, and do 3 million reads, with good
>     distribution, each
>     > node has served 1 million read requests.  If you increase to 6
>     nodes and
>     > keep RF=3, then each node now owns half as much data and serves only
>     > 500,000 reads.  Or more meaningfully in the same time it takes
>     to do 3
>     > million reads under the 3 node cluster you ought to be able to do 6
>     > million reads under the 6 node cluster since each node is just
>     > responsible for 1 million total reads.
>     >
>     Hi Eric,
>
>     I think I got your point.
>     In case of really evenly distributed  reads it may (or should?)
>     not make
>     any difference,
>
>     But when you do not distribute well the reads (and in that case only),
>     my understanding about RF was that it could help spreading the load :
>     In that case, with RF= 4 instead of 3,  with several clients
>     accessing keys
>     same key ranges, a coordinator could pick up one node to handle
>     the request
>     in 4 replicas instead of picking up one node in 3 , thus having
>     more "workers" to handle a request ?
>
>     Am I wrong here ?
>
>     Thank you for the clarification
>
>
>     --
>     best,
>     Alain
>

Re: How can I scale my read rate?

[Eric Stevens <mi...@gmail.com>]

Yes, throughput for a given partition key cannot be improved with
horizontal scaling.  You can increase RF to theoretically improve
throughput on that key, but actually in this case smart clients might hold
you back, because they're probably token aware, and will try to serve that
read off the key's primary replica, so all reads would be directed at a
single node for that key.

If you're reading at CL=QUORUM, there's a chance that increasing RF will
actually reduce performance rather than improve it, because you've
increased the total amount of work to serve the read (as well as the
write).  If you're reading at CL=ONE, increasing RF will increase the
chances of falling afoul of eventual consistency.

However that's not really a real-world scenario.  Or if it is, Cassandra is
probably the wrong tool to satisfy that kind of workload.

On Thu, Mar 23, 2017 at 11:43 PM Alain Rastoul <al...@gmail.com>
wrote:

On 24/03/2017 01:00, Eric Stevens wrote:
> Assuming an even distribution of data in your cluster, and an even
> distribution across those keys by your readers, you would not need to
> increase RF with cluster size to increase read performance.  If you have
> 3 nodes with RF=3, and do 3 million reads, with good distribution, each
> node has served 1 million read requests.  If you increase to 6 nodes and
> keep RF=3, then each node now owns half as much data and serves only
> 500,000 reads.  Or more meaningfully in the same time it takes to do 3
> million reads under the 3 node cluster you ought to be able to do 6
> million reads under the 6 node cluster since each node is just
> responsible for 1 million total reads.
>
Hi Eric,

I think I got your point.
In case of really evenly distributed  reads it may (or should?) not make
any difference,

But when you do not distribute well the reads (and in that case only),
my understanding about RF was that it could help spreading the load :
In that case, with RF= 4 instead of 3,  with several clients accessing keys
same key ranges, a coordinator could pick up one node to handle the request
in 4 replicas instead of picking up one node in 3 , thus having
more "workers" to handle a request ?

Am I wrong here ?

Thank you for the clarification


--
best,
Alain

Re: How can I scale my read rate?

[Anthony Grasso <an...@gmail.com>]

Keep in mind there are side effects to increasing to RF = 4

   - Storage requirements for each node will increase. Depending on the
   number of nodes in the cluster and the size of the data this could be
   significant.
   - Whilst the number of available coordinators increases, the number of
   nodes involved in QUORUM reads/writes will increase from 2 to 3.



On 24 March 2017 at 16:43, Alain Rastoul <al...@gmail.com> wrote:

> On 24/03/2017 01:00, Eric Stevens wrote:
>
>> Assuming an even distribution of data in your cluster, and an even
>> distribution across those keys by your readers, you would not need to
>> increase RF with cluster size to increase read performance.  If you have
>> 3 nodes with RF=3, and do 3 million reads, with good distribution, each
>> node has served 1 million read requests.  If you increase to 6 nodes and
>> keep RF=3, then each node now owns half as much data and serves only
>> 500,000 reads.  Or more meaningfully in the same time it takes to do 3
>> million reads under the 3 node cluster you ought to be able to do 6
>> million reads under the 6 node cluster since each node is just
>> responsible for 1 million total reads.
>>
>> Hi Eric,
>
> I think I got your point.
> In case of really evenly distributed  reads it may (or should?) not make
> any difference,
>
> But when you do not distribute well the reads (and in that case only),
> my understanding about RF was that it could help spreading the load :
> In that case, with RF= 4 instead of 3,  with several clients accessing keys
> same key ranges, a coordinator could pick up one node to handle the request
> in 4 replicas instead of picking up one node in 3 , thus having
> more "workers" to handle a request ?
>
> Am I wrong here ?
>
> Thank you for the clarification
>
>
> --
> best,
> Alain
>
>

Re: How can I scale my read rate?

[Alain Rastoul <al...@gmail.com>]

On 24/03/2017 01:00, Eric Stevens wrote:
> Assuming an even distribution of data in your cluster, and an even
> distribution across those keys by your readers, you would not need to
> increase RF with cluster size to increase read performance.  If you have
> 3 nodes with RF=3, and do 3 million reads, with good distribution, each
> node has served 1 million read requests.  If you increase to 6 nodes and
> keep RF=3, then each node now owns half as much data and serves only
> 500,000 reads.  Or more meaningfully in the same time it takes to do 3
> million reads under the 3 node cluster you ought to be able to do 6
> million reads under the 6 node cluster since each node is just
> responsible for 1 million total reads.
>
Hi Eric,

I think I got your point.
In case of really evenly distributed  reads it may (or should?) not make 
any difference,

But when you do not distribute well the reads (and in that case only),
my understanding about RF was that it could help spreading the load :
In that case, with RF= 4 instead of 3,  with several clients accessing keys
same key ranges, a coordinator could pick up one node to handle the request
in 4 replicas instead of picking up one node in 3 , thus having
more "workers" to handle a request ?

Am I wrong here ?

Thank you for the clarification


-- 
best,
Alain

Re: How can I scale my read rate?

[Eric Stevens <mi...@gmail.com>]

Assuming an even distribution of data in your cluster, and an even
distribution across those keys by your readers, you would not need to
increase RF with cluster size to increase read performance.  If you have 3
nodes with RF=3, and do 3 million reads, with good distribution, each node
has served 1 million read requests.  If you increase to 6 nodes and keep
RF=3, then each node now owns half as much data and serves only 500,000
reads.  Or more meaningfully in the same time it takes to do 3 million
reads under the 3 node cluster you ought to be able to do 6 million reads
under the 6 node cluster since each node is just responsible for 1 million
total reads.

On Mon, Mar 20, 2017 at 11:24 PM Alain Rastoul <al...@gmail.com>
wrote:

> On 20/03/2017 22:05, Michael Wojcikiewicz wrote:
> > Not sure if someone has suggested this, but I believe it's not
> > sufficient to simply add nodes to a cluster to increase read
> > performance: you also need to alter the ReplicationFactor of the
> > keyspace to a larger value as you increase your cluster gets larger.
> >
> > ie. data is available from more nodes in the cluster for each query.
> >
> Yes, good point in case of cluster growth, there would be more replica
> to handle same key ranges.
> And also readjust token ranges :
> https://cassandra.apache.org/doc/latest/operating/topo_changes.html
>
> SG, can you give some information (or share your code) about how you
> generate your data and how you read it ?
>
> --
> best,
> Alain
>
>

Re: How can I scale my read rate?

[Alain Rastoul <al...@gmail.com>]

On 20/03/2017 22:05, Michael Wojcikiewicz wrote:
> Not sure if someone has suggested this, but I believe it's not
> sufficient to simply add nodes to a cluster to increase read
> performance: you also need to alter the ReplicationFactor of the
> keyspace to a larger value as you increase your cluster gets larger.
>
> ie. data is available from more nodes in the cluster for each query.
>
Yes, good point in case of cluster growth, there would be more replica 
to handle same key ranges.
And also readjust token ranges :
https://cassandra.apache.org/doc/latest/operating/topo_changes.html

SG, can you give some information (or share your code) about how you 
generate your data and how you read it ?

-- 
best,
Alain

Re: How can I scale my read rate?

[Michael Wojcikiewicz <mi...@withkash.com>]

Not sure if someone has suggested this, but I believe it's not sufficient
to simply add nodes to a cluster to increase read performance: you also
need to alter the ReplicationFactor of the keyspace to a larger value as
you increase your cluster gets larger.

ie. data is available from more nodes in the cluster for each query.

On Mon, Mar 20, 2017 at 4:55 AM, Arvydas Jonusonis <
arvydas.jonusonis@gmail.com> wrote:

> re: (1)
> You can do enable it in the *cqlsh* by executing TRACING ON;
> Then each time you run a query, you will get a summary of the timing info.
>
> You can also enable it directly in the driver and get the results
> programmatically: https://academy.datastax.com/resources/datastax-java-
> driver-request-tracing
>
> Arvydas
>
> On Mon, Mar 20, 2017 at 2:35 AM, S G <sg...@gmail.com> wrote:
>
>> I am not using prepared statements. There are two reasons for that:
>>
>> 1) https://issues.apache.org/jira/browse/CASSANDRA-3634 tells me that
>> the performance improvements with prepared statements are capped at about
>> 20% - so I would not see a drastic difference.
>> 2) https://docs.datastax.com/en/developer/java-driver/3.1/ma
>> nual/statements/prepared/ tells me to avoid preparing select queries if
>> I expect a change of columns in my table down the road.
>>
>> So I wasn't very keen on trying those out.
>> Please suggest if I can expect to get more than 20% performance using the
>> same and I will try that too.
>>
>>
>> I did some more testing to see if my client machines were the bottleneck.
>> For a 6-node Cassandra cluster (each VM having 8-cores), I got 26,000
>> reads/sec for all of the following:
>> 1) Client nodes:1, Threads: 60
>> 2) Client nodes:3, Threads: 180
>> 3) Client nodes:5, Threads: 300
>> 4) Client nodes:10, Threads: 600
>> 5) Client nodes:20, Threads: 1200
>>
>> So adding more client nodes or threads to those client nodes is not
>> having any effect.
>> I am suspecting Cassandra is simply not allowing me to go any further.
>>
>> Reads/second is measured both by the opscenter and by my own logic in the
>> client machines code. They seem to agree more or less.
>>
>> Primary keys for my schema are:
>>     PRIMARY KEY((name, phone), age)
>> name: text
>> phone: int
>> age: int
>>
>> And there are very few columns in the table apart from the above. None of
>> them is more than few bytes long.
>>
>> 1) Does Cassandra have something like debug=timing
>> <http://stackoverflow.com/questions/14712175/splitting-solr-response-time>
>> in Solr which gives all the information about where query-time was spent?
>> 2) Does the nodetool have something that can give me some hint?
>> 3) Is there a JMX metrics I should look for?
>>
>>
>> Thanks
>> SG
>>
>>
>>
>>
>>
>> On Sun, Mar 19, 2017 at 5:03 AM, James Carman <james@carmanconsulting.com
>> > wrote:
>>
>>> Have you tried using PreparedStatements?
>>>
>>> On Sat, Mar 18, 2017 at 9:47 PM S G <sg...@gmail.com> wrote:
>>>
>>>> ok, I gave the executeAsync() a try.
>>>> Good part is that it was really easy to write the code for that.
>>>> Bad part is that it did not had a huge effect on my throughput - I
>>>> gained about 5% increase in throughput.
>>>> I suspect it is so because my queries are all get-by-primary-key
>>>> queries and were anyways completing in less than 2 milliseconds.
>>>> So there was not much wait to begin with.
>>>>
>>>>
>>>> Here is my code:
>>>>
>>>> String getByKeyQueryStr = "Select * from fooTable where key = " + key;
>>>> //ResultSet result = session.execute(getByKeyQueryStr);  // Previous
>>>> code
>>>> ResultSetFuture future = session.executeAsync(getByKeyQueryStr);
>>>> FutureCallback<ResultSet> callback = new MyFutureCallback();
>>>> executor = MoreExecutors.sameThreadExecutor();
>>>> //executor = Executors.newFixedThreadPool(3); // Tried this too, no
>>>> effect
>>>> //executor = Executors.newFixedThreadPool(10); // Tried this too, no
>>>> effect
>>>> Futures.addCallback(future, callback, executor);
>>>>
>>>> Can I improve the above code in some way?
>>>> Are there any JMX metrics that can tell me what's going on?
>>>>
>>>> From the vmstat command, I see that CPU idle time is about 70% even
>>>> though I am running about 60 threads per VM
>>>> Total 20 client-VMs with 8 cores each are querying a Cassandra cluster
>>>> with 16 VMs, 8-core each too.
>>>>
>>>> [image: Screen Shot 2017-03-18 at 6.46.03 PM.png]
>>>> ​
>>>> ​
>>>>
>>>>
>>>> Thanks
>>>> SG
>>>>
>>>>
>>>> On Sat, Mar 18, 2017 at 5:38 PM, S G <sg...@gmail.com> wrote:
>>>>
>>>> Thanks. It seems that you guys have found executeAsync to yield good
>>>> results.
>>>> I want to share my understanding how this could benefit performance and
>>>> some validation from the group will be awesome.
>>>>
>>>> I will call executeAsync() each time I want to get by primary-key.
>>>> That ways, my client thread is not blocked anymore and I can submit a
>>>> lot more requests per unit time.
>>>> The async requests get piled on the underlying Netty I/O thread which
>>>> ensures that it is always busy all the time.
>>>> Earlier, the Netty I/O thread would have wasted some cycles when the
>>>> sync-execute method was processing the results.
>>>> And earlier, the client thread would also have wasted some cycles
>>>> waiting for netty-thread to complete.
>>>>
>>>> With executeAsync(), none of them is waiting.
>>>> Only thing to ensure is that the Netty thread's queue does not grow
>>>> indefinitely.
>>>>
>>>> If the above theory is correct, then it sounds like a really good thing
>>>> to try.
>>>> If not, please do share some more details.
>>>>
>>>>
>>>>
>>>>
>>>> On Sat, Mar 18, 2017 at 2:00 PM, <j....@enercast.de> wrote:
>>>>
>>>> +1 for executeAsync – had a long time to argue that it’s not bad as
>>>> with good old rdbms.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Gesendet von meinem Windows 10 Phone
>>>>
>>>>
>>>>
>>>> *Von: *Arvydas Jonusonis <ar...@gmail.com>
>>>> *Gesendet: *Samstag, 18. März 2017 19:08
>>>> *An: *user@cassandra.apache.org
>>>> *Betreff: *Re: How can I scale my read rate?
>>>>
>>>>
>>>>
>>>> ..then you're not taking advantage of request pipelining. Use
>>>> executeAsync - this will increase your throughput for sure.
>>>>
>>>>
>>>>
>>>> http://www.datastax.com/dev/blog/java-driver-async-queries
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Sat, Mar 18, 2017 at 08:00 S G <sg...@gmail.com> wrote:
>>>>
>>>> I have enabled JMX but not sure what metrics to look for - they are way
>>>> too many of them.
>>>>
>>>> I am using session.execute(...)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Mar 17, 2017 at 2:07 PM, Arvydas Jonusonis <
>>>> arvydas.jonusonis@gmail.com> wrote:
>>>>
>>>> It would be interesting to see some of the driver metrics (in your
>>>> stress test tool) - if you enable JMX, they should be exposed by default.
>>>>
>>>> Also, are you using session.execute(..) or session.executeAsync(..) ?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>
>


-- 
--mike

Re: How can I scale my read rate?

[Arvydas Jonusonis <ar...@gmail.com>]

re: (1)
You can do enable it in the *cqlsh* by executing TRACING ON;
Then each time you run a query, you will get a summary of the timing info.

You can also enable it directly in the driver and get the results
programmatically:
https://academy.datastax.com/resources/datastax-java-driver-request-tracing

Arvydas

On Mon, Mar 20, 2017 at 2:35 AM, S G <sg...@gmail.com> wrote:

> I am not using prepared statements. There are two reasons for that:
>
> 1) https://issues.apache.org/jira/browse/CASSANDRA-3634 tells me that the
> performance improvements with prepared statements are capped at about 20% -
> so I would not see a drastic difference.
> 2) https://docs.datastax.com/en/developer/java-driver/3.1/
> manual/statements/prepared/ tells me to avoid preparing select queries if
> I expect a change of columns in my table down the road.
>
> So I wasn't very keen on trying those out.
> Please suggest if I can expect to get more than 20% performance using the
> same and I will try that too.
>
>
> I did some more testing to see if my client machines were the bottleneck.
> For a 6-node Cassandra cluster (each VM having 8-cores), I got 26,000
> reads/sec for all of the following:
> 1) Client nodes:1, Threads: 60
> 2) Client nodes:3, Threads: 180
> 3) Client nodes:5, Threads: 300
> 4) Client nodes:10, Threads: 600
> 5) Client nodes:20, Threads: 1200
>
> So adding more client nodes or threads to those client nodes is not having
> any effect.
> I am suspecting Cassandra is simply not allowing me to go any further.
>
> Reads/second is measured both by the opscenter and by my own logic in the
> client machines code. They seem to agree more or less.
>
> Primary keys for my schema are:
>     PRIMARY KEY((name, phone), age)
> name: text
> phone: int
> age: int
>
> And there are very few columns in the table apart from the above. None of
> them is more than few bytes long.
>
> 1) Does Cassandra have something like debug=timing
> <http://stackoverflow.com/questions/14712175/splitting-solr-response-time>
> in Solr which gives all the information about where query-time was spent?
> 2) Does the nodetool have something that can give me some hint?
> 3) Is there a JMX metrics I should look for?
>
>
> Thanks
> SG
>
>
>
>
>
> On Sun, Mar 19, 2017 at 5:03 AM, James Carman <ja...@carmanconsulting.com>
> wrote:
>
>> Have you tried using PreparedStatements?
>>
>> On Sat, Mar 18, 2017 at 9:47 PM S G <sg...@gmail.com> wrote:
>>
>>> ok, I gave the executeAsync() a try.
>>> Good part is that it was really easy to write the code for that.
>>> Bad part is that it did not had a huge effect on my throughput - I
>>> gained about 5% increase in throughput.
>>> I suspect it is so because my queries are all get-by-primary-key queries
>>> and were anyways completing in less than 2 milliseconds.
>>> So there was not much wait to begin with.
>>>
>>>
>>> Here is my code:
>>>
>>> String getByKeyQueryStr = "Select * from fooTable where key = " + key;
>>> //ResultSet result = session.execute(getByKeyQueryStr);  // Previous
>>> code
>>> ResultSetFuture future = session.executeAsync(getByKeyQueryStr);
>>> FutureCallback<ResultSet> callback = new MyFutureCallback();
>>> executor = MoreExecutors.sameThreadExecutor();
>>> //executor = Executors.newFixedThreadPool(3); // Tried this too, no
>>> effect
>>> //executor = Executors.newFixedThreadPool(10); // Tried this too, no
>>> effect
>>> Futures.addCallback(future, callback, executor);
>>>
>>> Can I improve the above code in some way?
>>> Are there any JMX metrics that can tell me what's going on?
>>>
>>> From the vmstat command, I see that CPU idle time is about 70% even
>>> though I am running about 60 threads per VM
>>> Total 20 client-VMs with 8 cores each are querying a Cassandra cluster
>>> with 16 VMs, 8-core each too.
>>>
>>> [image: Screen Shot 2017-03-18 at 6.46.03 PM.png]
>>> ​
>>> ​
>>>
>>>
>>> Thanks
>>> SG
>>>
>>>
>>> On Sat, Mar 18, 2017 at 5:38 PM, S G <sg...@gmail.com> wrote:
>>>
>>> Thanks. It seems that you guys have found executeAsync to yield good
>>> results.
>>> I want to share my understanding how this could benefit performance and
>>> some validation from the group will be awesome.
>>>
>>> I will call executeAsync() each time I want to get by primary-key.
>>> That ways, my client thread is not blocked anymore and I can submit a
>>> lot more requests per unit time.
>>> The async requests get piled on the underlying Netty I/O thread which
>>> ensures that it is always busy all the time.
>>> Earlier, the Netty I/O thread would have wasted some cycles when the
>>> sync-execute method was processing the results.
>>> And earlier, the client thread would also have wasted some cycles
>>> waiting for netty-thread to complete.
>>>
>>> With executeAsync(), none of them is waiting.
>>> Only thing to ensure is that the Netty thread's queue does not grow
>>> indefinitely.
>>>
>>> If the above theory is correct, then it sounds like a really good thing
>>> to try.
>>> If not, please do share some more details.
>>>
>>>
>>>
>>>
>>> On Sat, Mar 18, 2017 at 2:00 PM, <j....@enercast.de> wrote:
>>>
>>> +1 for executeAsync – had a long time to argue that it’s not bad as with
>>> good old rdbms.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Gesendet von meinem Windows 10 Phone
>>>
>>>
>>>
>>> *Von: *Arvydas Jonusonis <ar...@gmail.com>
>>> *Gesendet: *Samstag, 18. März 2017 19:08
>>> *An: *user@cassandra.apache.org
>>> *Betreff: *Re: How can I scale my read rate?
>>>
>>>
>>>
>>> ..then you're not taking advantage of request pipelining. Use
>>> executeAsync - this will increase your throughput for sure.
>>>
>>>
>>>
>>> http://www.datastax.com/dev/blog/java-driver-async-queries
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Mar 18, 2017 at 08:00 S G <sg...@gmail.com> wrote:
>>>
>>> I have enabled JMX but not sure what metrics to look for - they are way
>>> too many of them.
>>>
>>> I am using session.execute(...)
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Mar 17, 2017 at 2:07 PM, Arvydas Jonusonis <
>>> arvydas.jonusonis@gmail.com> wrote:
>>>
>>> It would be interesting to see some of the driver metrics (in your
>>> stress test tool) - if you enable JMX, they should be exposed by default.
>>>
>>> Also, are you using session.execute(..) or session.executeAsync(..) ?
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>

Re: How can I scale my read rate?

[S G <sg...@gmail.com>]

I am not using prepared statements. There are two reasons for that:

1) https://issues.apache.org/jira/browse/CASSANDRA-3634 tells me that the
performance improvements with prepared statements are capped at about 20% -
so I would not see a drastic difference.
2)
https://docs.datastax.com/en/developer/java-driver/3.1/manual/statements/prepared/
tells me to avoid preparing select queries if I expect a change of columns
in my table down the road.

So I wasn't very keen on trying those out.
Please suggest if I can expect to get more than 20% performance using the
same and I will try that too.


I did some more testing to see if my client machines were the bottleneck.
For a 6-node Cassandra cluster (each VM having 8-cores), I got 26,000
reads/sec for all of the following:
1) Client nodes:1, Threads: 60
2) Client nodes:3, Threads: 180
3) Client nodes:5, Threads: 300
4) Client nodes:10, Threads: 600
5) Client nodes:20, Threads: 1200

So adding more client nodes or threads to those client nodes is not having
any effect.
I am suspecting Cassandra is simply not allowing me to go any further.

Reads/second is measured both by the opscenter and by my own logic in the
client machines code. They seem to agree more or less.

Primary keys for my schema are:
    PRIMARY KEY((name, phone), age)
name: text
phone: int
age: int

And there are very few columns in the table apart from the above. None of
them is more than few bytes long.

1) Does Cassandra have something like debug=timing
<http://stackoverflow.com/questions/14712175/splitting-solr-response-time>
in Solr which gives all the information about where query-time was spent?
2) Does the nodetool have something that can give me some hint?
3) Is there a JMX metrics I should look for?


Thanks
SG





On Sun, Mar 19, 2017 at 5:03 AM, James Carman <ja...@carmanconsulting.com>
wrote:

> Have you tried using PreparedStatements?
>
> On Sat, Mar 18, 2017 at 9:47 PM S G <sg...@gmail.com> wrote:
>
>> ok, I gave the executeAsync() a try.
>> Good part is that it was really easy to write the code for that.
>> Bad part is that it did not had a huge effect on my throughput - I gained
>> about 5% increase in throughput.
>> I suspect it is so because my queries are all get-by-primary-key queries
>> and were anyways completing in less than 2 milliseconds.
>> So there was not much wait to begin with.
>>
>>
>> Here is my code:
>>
>> String getByKeyQueryStr = "Select * from fooTable where key = " + key;
>> //ResultSet result = session.execute(getByKeyQueryStr);  // Previous code
>> ResultSetFuture future = session.executeAsync(getByKeyQueryStr);
>> FutureCallback<ResultSet> callback = new MyFutureCallback();
>> executor = MoreExecutors.sameThreadExecutor();
>> //executor = Executors.newFixedThreadPool(3); // Tried this too, no
>> effect
>> //executor = Executors.newFixedThreadPool(10); // Tried this too, no
>> effect
>> Futures.addCallback(future, callback, executor);
>>
>> Can I improve the above code in some way?
>> Are there any JMX metrics that can tell me what's going on?
>>
>> From the vmstat command, I see that CPU idle time is about 70% even
>> though I am running about 60 threads per VM
>> Total 20 client-VMs with 8 cores each are querying a Cassandra cluster
>> with 16 VMs, 8-core each too.
>>
>> [image: Screen Shot 2017-03-18 at 6.46.03 PM.png]
>> ​
>> ​
>>
>>
>> Thanks
>> SG
>>
>>
>> On Sat, Mar 18, 2017 at 5:38 PM, S G <sg...@gmail.com> wrote:
>>
>> Thanks. It seems that you guys have found executeAsync to yield good
>> results.
>> I want to share my understanding how this could benefit performance and
>> some validation from the group will be awesome.
>>
>> I will call executeAsync() each time I want to get by primary-key.
>> That ways, my client thread is not blocked anymore and I can submit a lot
>> more requests per unit time.
>> The async requests get piled on the underlying Netty I/O thread which
>> ensures that it is always busy all the time.
>> Earlier, the Netty I/O thread would have wasted some cycles when the
>> sync-execute method was processing the results.
>> And earlier, the client thread would also have wasted some cycles waiting
>> for netty-thread to complete.
>>
>> With executeAsync(), none of them is waiting.
>> Only thing to ensure is that the Netty thread's queue does not grow
>> indefinitely.
>>
>> If the above theory is correct, then it sounds like a really good thing
>> to try.
>> If not, please do share some more details.
>>
>>
>>
>>
>> On Sat, Mar 18, 2017 at 2:00 PM, <j....@enercast.de> wrote:
>>
>> +1 for executeAsync – had a long time to argue that it’s not bad as with
>> good old rdbms.
>>
>>
>>
>>
>>
>>
>>
>> Gesendet von meinem Windows 10 Phone
>>
>>
>>
>> *Von: *Arvydas Jonusonis <ar...@gmail.com>
>> *Gesendet: *Samstag, 18. März 2017 19:08
>> *An: *user@cassandra.apache.org
>> *Betreff: *Re: How can I scale my read rate?
>>
>>
>>
>> ..then you're not taking advantage of request pipelining. Use
>> executeAsync - this will increase your throughput for sure.
>>
>>
>>
>> http://www.datastax.com/dev/blog/java-driver-async-queries
>>
>>
>>
>>
>>
>> On Sat, Mar 18, 2017 at 08:00 S G <sg...@gmail.com> wrote:
>>
>> I have enabled JMX but not sure what metrics to look for - they are way
>> too many of them.
>>
>> I am using session.execute(...)
>>
>>
>>
>>
>>
>> On Fri, Mar 17, 2017 at 2:07 PM, Arvydas Jonusonis <
>> arvydas.jonusonis@gmail.com> wrote:
>>
>> It would be interesting to see some of the driver metrics (in your stress
>> test tool) - if you enable JMX, they should be exposed by default.
>>
>> Also, are you using session.execute(..) or session.executeAsync(..) ?
>>
>>
>>
>>
>>
>>
>>

Re: How can I scale my read rate?

[James Carman <ja...@carmanconsulting.com>]

Have you tried using PreparedStatements?

On Sat, Mar 18, 2017 at 9:47 PM S G <sg...@gmail.com> wrote:

> ok, I gave the executeAsync() a try.
> Good part is that it was really easy to write the code for that.
> Bad part is that it did not had a huge effect on my throughput - I gained
> about 5% increase in throughput.
> I suspect it is so because my queries are all get-by-primary-key queries
> and were anyways completing in less than 2 milliseconds.
> So there was not much wait to begin with.
>
>
> Here is my code:
>
> String getByKeyQueryStr = "Select * from fooTable where key = " + key;
> //ResultSet result = session.execute(getByKeyQueryStr);  // Previous code
> ResultSetFuture future = session.executeAsync(getByKeyQueryStr);
> FutureCallback<ResultSet> callback = new MyFutureCallback();
> executor = MoreExecutors.sameThreadExecutor();
> //executor = Executors.newFixedThreadPool(3); // Tried this too, no effect
> //executor = Executors.newFixedThreadPool(10); // Tried this too, no effect
> Futures.addCallback(future, callback, executor);
>
> Can I improve the above code in some way?
> Are there any JMX metrics that can tell me what's going on?
>
> From the vmstat command, I see that CPU idle time is about 70% even though
> I am running about 60 threads per VM
> Total 20 client-VMs with 8 cores each are querying a Cassandra cluster
> with 16 VMs, 8-core each too.
>
> [image: Screen Shot 2017-03-18 at 6.46.03 PM.png]
> ​
> ​
>
>
> Thanks
> SG
>
>
> On Sat, Mar 18, 2017 at 5:38 PM, S G <sg...@gmail.com> wrote:
>
> Thanks. It seems that you guys have found executeAsync to yield good
> results.
> I want to share my understanding how this could benefit performance and
> some validation from the group will be awesome.
>
> I will call executeAsync() each time I want to get by primary-key.
> That ways, my client thread is not blocked anymore and I can submit a lot
> more requests per unit time.
> The async requests get piled on the underlying Netty I/O thread which
> ensures that it is always busy all the time.
> Earlier, the Netty I/O thread would have wasted some cycles when the
> sync-execute method was processing the results.
> And earlier, the client thread would also have wasted some cycles waiting
> for netty-thread to complete.
>
> With executeAsync(), none of them is waiting.
> Only thing to ensure is that the Netty thread's queue does not grow
> indefinitely.
>
> If the above theory is correct, then it sounds like a really good thing to
> try.
> If not, please do share some more details.
>
>
>
>
> On Sat, Mar 18, 2017 at 2:00 PM, <j....@enercast.de> wrote:
>
> +1 for executeAsync – had a long time to argue that it’s not bad as with
> good old rdbms.
>
>
>
>
>
>
>
> Gesendet von meinem Windows 10 Phone
>
>
>
> *Von: *Arvydas Jonusonis <ar...@gmail.com>
> *Gesendet: *Samstag, 18. März 2017 19:08
> *An: *user@cassandra.apache.org
> *Betreff: *Re: How can I scale my read rate?
>
>
>
> ..then you're not taking advantage of request pipelining. Use executeAsync
> - this will increase your throughput for sure.
>
>
>
> http://www.datastax.com/dev/blog/java-driver-async-queries
>
>
>
>
>
> On Sat, Mar 18, 2017 at 08:00 S G <sg...@gmail.com> wrote:
>
> I have enabled JMX but not sure what metrics to look for - they are way
> too many of them.
>
> I am using session.execute(...)
>
>
>
>
>
> On Fri, Mar 17, 2017 at 2:07 PM, Arvydas Jonusonis <
> arvydas.jonusonis@gmail.com> wrote:
>
> It would be interesting to see some of the driver metrics (in your stress
> test tool) - if you enable JMX, they should be exposed by default.
>
> Also, are you using session.execute(..) or session.executeAsync(..) ?
>
>
>
>
>
>
>

Re: How can I scale my read rate?

[S G <sg...@gmail.com>]

ok, I gave the executeAsync() a try.
Good part is that it was really easy to write the code for that.
Bad part is that it did not had a huge effect on my throughput - I gained
about 5% increase in throughput.
I suspect it is so because my queries are all get-by-primary-key queries
and were anyways completing in less than 2 milliseconds.
So there was not much wait to begin with.


Here is my code:

String getByKeyQueryStr = "Select * from fooTable where key = " + key;
//ResultSet result = session.execute(getByKeyQueryStr);  // Previous code
ResultSetFuture future = session.executeAsync(getByKeyQueryStr);
FutureCallback<ResultSet> callback = new MyFutureCallback();
executor = MoreExecutors.sameThreadExecutor();
//executor = Executors.newFixedThreadPool(3); // Tried this too, no effect
//executor = Executors.newFixedThreadPool(10); // Tried this too, no effect
Futures.addCallback(future, callback, executor);

Can I improve the above code in some way?
Are there any JMX metrics that can tell me what's going on?

From the vmstat command, I see that CPU idle time is about 70% even though
I am running about 60 threads per VM
Total 20 client-VMs with 8 cores each are querying a Cassandra cluster with
16 VMs, 8-core each too.


​
​


Thanks
SG


On Sat, Mar 18, 2017 at 5:38 PM, S G <sg...@gmail.com> wrote:

> Thanks. It seems that you guys have found executeAsync to yield good
> results.
> I want to share my understanding how this could benefit performance and
> some validation from the group will be awesome.
>
> I will call executeAsync() each time I want to get by primary-key.
> That ways, my client thread is not blocked anymore and I can submit a lot
> more requests per unit time.
> The async requests get piled on the underlying Netty I/O thread which
> ensures that it is always busy all the time.
> Earlier, the Netty I/O thread would have wasted some cycles when the
> sync-execute method was processing the results.
> And earlier, the client thread would also have wasted some cycles waiting
> for netty-thread to complete.
>
> With executeAsync(), none of them is waiting.
> Only thing to ensure is that the Netty thread's queue does not grow
> indefinitely.
>
> If the above theory is correct, then it sounds like a really good thing to
> try.
> If not, please do share some more details.
>
>
>
>
> On Sat, Mar 18, 2017 at 2:00 PM, <j....@enercast.de> wrote:
>
>> +1 for executeAsync – had a long time to argue that it’s not bad as with
>> good old rdbms.
>>
>>
>>
>>
>>
>>
>>
>> Gesendet von meinem Windows 10 Phone
>>
>>
>>
>> *Von: *Arvydas Jonusonis <ar...@gmail.com>
>> *Gesendet: *Samstag, 18. März 2017 19:08
>> *An: *user@cassandra.apache.org
>> *Betreff: *Re: How can I scale my read rate?
>>
>>
>>
>> ..then you're not taking advantage of request pipelining. Use
>> executeAsync - this will increase your throughput for sure.
>>
>>
>>
>> http://www.datastax.com/dev/blog/java-driver-async-queries
>>
>>
>>
>>
>>
>> On Sat, Mar 18, 2017 at 08:00 S G <sg...@gmail.com> wrote:
>>
>> I have enabled JMX but not sure what metrics to look for - they are way
>> too many of them.
>>
>> I am using session.execute(...)
>>
>>
>>
>>
>>
>> On Fri, Mar 17, 2017 at 2:07 PM, Arvydas Jonusonis <
>> arvydas.jonusonis@gmail.com> wrote:
>>
>> It would be interesting to see some of the driver metrics (in your stress
>> test tool) - if you enable JMX, they should be exposed by default.
>>
>> Also, are you using session.execute(..) or session.executeAsync(..) ?
>>
>>
>>
>>
>>
>

Re: How can I scale my read rate?

[S G <sg...@gmail.com>]

Thanks. It seems that you guys have found executeAsync to yield good
results.
I want to share my understanding how this could benefit performance and
some validation from the group will be awesome.

I will call executeAsync() each time I want to get by primary-key.
That ways, my client thread is not blocked anymore and I can submit a lot
more requests per unit time.
The async requests get piled on the underlying Netty I/O thread which
ensures that it is always busy all the time.
Earlier, the Netty I/O thread would have wasted some cycles when the
sync-execute method was processing the results.
And earlier, the client thread would also have wasted some cycles waiting
for netty-thread to complete.

With executeAsync(), none of them is waiting.
Only thing to ensure is that the Netty thread's queue does not grow
indefinitely.

If the above theory is correct, then it sounds like a really good thing to
try.
If not, please do share some more details.




On Sat, Mar 18, 2017 at 2:00 PM, <j....@enercast.de> wrote:

> +1 for executeAsync – had a long time to argue that it’s not bad as with
> good old rdbms.
>
>
>
>
>
>
>
> Gesendet von meinem Windows 10 Phone
>
>
>
> *Von: *Arvydas Jonusonis <ar...@gmail.com>
> *Gesendet: *Samstag, 18. März 2017 19:08
> *An: *user@cassandra.apache.org
> *Betreff: *Re: How can I scale my read rate?
>
>
>
> ..then you're not taking advantage of request pipelining. Use executeAsync
> - this will increase your throughput for sure.
>
>
>
> http://www.datastax.com/dev/blog/java-driver-async-queries
>
>
>
>
>
> On Sat, Mar 18, 2017 at 08:00 S G <sg...@gmail.com> wrote:
>
> I have enabled JMX but not sure what metrics to look for - they are way
> too many of them.
>
> I am using session.execute(...)
>
>
>
>
>
> On Fri, Mar 17, 2017 at 2:07 PM, Arvydas Jonusonis <
> arvydas.jonusonis@gmail.com> wrote:
>
> It would be interesting to see some of the driver metrics (in your stress
> test tool) - if you enable JMX, they should be exposed by default.
>
> Also, are you using session.execute(..) or session.executeAsync(..) ?
>
>
>
>
>

Re: How can I scale my read rate?

[Arvydas Jonusonis <ar...@gmail.com>]

..then you're not taking advantage of request pipelining. Use executeAsync
- this will increase your throughput for sure.

http://www.datastax.com/dev/blog/java-driver-async-queries


On Sat, Mar 18, 2017 at 08:00 S G <sg...@gmail.com> wrote:

> I have enabled JMX but not sure what metrics to look for - they are way
> too many of them.
> I am using session.execute(...)
>
>
> On Fri, Mar 17, 2017 at 2:07 PM, Arvydas Jonusonis <
> arvydas.jonusonis@gmail.com> wrote:
>
> It would be interesting to see some of the driver metrics (in your stress
> test tool) - if you enable JMX, they should be exposed by default.
>
> Also, are you using session.execute(..) or session.executeAsync(..) ?
>
> Arvydas
>
> On Thu, Mar 16, 2017 at 00:44 Eric Stevens <mi...@gmail.com> wrote:
>
> I assume your graphs are from a node on the Cassandra cluster.  You're
> really not stressing out that cluster, load average under 1 is an extremely
> bored cluster wishing you would give it something to work on.  I have a
> hunch that the limiting factor might be in your read stress testing tool.
>
> Are you doing sequential reads where each thread waits for the previous
> read to complete before attempting the next read?  You might be spending
> most of your time just waiting for network round trips.  I would suggest
> slowly ramping up the number of simultaneous outstanding reads per thread
> until you start to get a non-trivial amount of read timeout exceptions - at
> that point is where you've hit your maximum read throughput.  Depending on
> what kind of disks you have there you might expect per-node load averages
> to be in the 30+ range (maybe lower, that heap size is a bit tight).
>
> On Wed, Mar 15, 2017 at 4:22 PM Jonathan Haddad <jo...@jonhaddad.com> wrote:
>
> Really no need to be rude.  Why the hostility?
>
> On Wed, Mar 15, 2017 at 2:12 PM daemeon reiydelle <da...@gmail.com>
> wrote:
>
> with that brain damaged memory setting yes.
>
>
> *.......*
>
>
>
> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 <(415)%20501-0198>London
> (+44) (0) 20 8144 9872 <+44%2020%208144%209872>*
>
> On Tue, Mar 14, 2017 at 3:45 PM, S G <sg...@gmail.com> wrote:
>
> Hi,
>
> I am trying to scale my read throughput.
>
> I have a 12 node Cassandra cluster, each node has:
> RAM: 5.5gb
> Disk: 64gb
> C* version: 3.3
> Java: 1.8.0_51
>
> The cluster has 2 tables - each with 2 million rows.
> Partition keys for all these rows are unique.
>
> I am stress-reading it from a 15-node client cluster.
> Each read-client has 40 threads, so total 600 read-threads from 12
> machines.
> Each read query is get-by-primary-key where keys are read randomly from a
> file having all primary-keys.
>
>
> I am able to get only 15,000 reads/second from the entire system.
> Is this the best read performance I can expect?
> Are there any benchmarks I can compare against?
>
>
> I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching to
> {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
>
> I have also captured few JMX metric graphs as follows:
>
> ​
> Garbage collection metrics did not look very interesting, hence I have not
> added the same.
>
> Any ideas what more I can do to debug the system and increase its read
> throughput?
>
> Thanks
> SG
>
>
>
> On Sat, Mar 11, 2017 at 2:33 PM, Jeff Jirsa <jj...@gmail.com> wrote:
>
> 5.5G of ram isn't very much for a jvm based database. Having fewer
> instances with more ram will likely give you better performance.
>
> Also, this is probably a better question for the user list than the Dev
> list
>
> --
> Jeff Jirsa
>
>
> > On Mar 11, 2017, at 1:20 PM, S G <sg...@gmail.com> wrote:
> >
> > Hi,
> >
> > I have a 9 node cassandra cluster, each node has:
> > RAM: 5.5gb
> > Disk: 64gb
> > C* version: 3.3
> > Java: 1.8.0_51
> >
> > The cluster stores about 2 million rows and partition keys for all these
> > rows are unique.
> >
> > I am stress-reading it from a 12-node client cluster.
> > Each read-client has 50 threads, so total 600 read-threads from 12
> machines.
> > Each read query is get-by-primary-key where keys are read randomly from a
> > file having all primary-keys.
> >
> >
> > I am able to get only 15,000 reads/second from the entire system.
> > Is this the best read performance I can expect?
> > Are there any benchmarks I can compare against?
> >
> >
> > I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching to
> > {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
> >
> > Thx,
> > SG
>
>
>
>
>

Re: How can I scale my read rate?

[S G <sg...@gmail.com>]

I have enabled JMX but not sure what metrics to look for - they are way too
many of them.
I am using session.execute(...)


On Fri, Mar 17, 2017 at 2:07 PM, Arvydas Jonusonis <
arvydas.jonusonis@gmail.com> wrote:

> It would be interesting to see some of the driver metrics (in your stress
> test tool) - if you enable JMX, they should be exposed by default.
>
> Also, are you using session.execute(..) or session.executeAsync(..) ?
>
> Arvydas
>
> On Thu, Mar 16, 2017 at 00:44 Eric Stevens <mi...@gmail.com> wrote:
>
>> I assume your graphs are from a node on the Cassandra cluster.  You're
>> really not stressing out that cluster, load average under 1 is an extremely
>> bored cluster wishing you would give it something to work on.  I have a
>> hunch that the limiting factor might be in your read stress testing tool.
>>
>> Are you doing sequential reads where each thread waits for the previous
>> read to complete before attempting the next read?  You might be spending
>> most of your time just waiting for network round trips.  I would suggest
>> slowly ramping up the number of simultaneous outstanding reads per thread
>> until you start to get a non-trivial amount of read timeout exceptions - at
>> that point is where you've hit your maximum read throughput.  Depending on
>> what kind of disks you have there you might expect per-node load averages
>> to be in the 30+ range (maybe lower, that heap size is a bit tight).
>>
>> On Wed, Mar 15, 2017 at 4:22 PM Jonathan Haddad <jo...@jonhaddad.com>
>> wrote:
>>
>> Really no need to be rude.  Why the hostility?
>>
>> On Wed, Mar 15, 2017 at 2:12 PM daemeon reiydelle <da...@gmail.com>
>> wrote:
>>
>> with that brain damaged memory setting yes.
>>
>>
>> *.......*
>>
>>
>>
>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 <(415)%20501-0198>London
>> (+44) (0) 20 8144 9872 <+44%2020%208144%209872>*
>>
>> On Tue, Mar 14, 2017 at 3:45 PM, S G <sg...@gmail.com> wrote:
>>
>> Hi,
>>
>> I am trying to scale my read throughput.
>>
>> I have a 12 node Cassandra cluster, each node has:
>> RAM: 5.5gb
>> Disk: 64gb
>> C* version: 3.3
>> Java: 1.8.0_51
>>
>> The cluster has 2 tables - each with 2 million rows.
>> Partition keys for all these rows are unique.
>>
>> I am stress-reading it from a 15-node client cluster.
>> Each read-client has 40 threads, so total 600 read-threads from 12
>> machines.
>> Each read query is get-by-primary-key where keys are read randomly from a
>> file having all primary-keys.
>>
>>
>> I am able to get only 15,000 reads/second from the entire system.
>> Is this the best read performance I can expect?
>> Are there any benchmarks I can compare against?
>>
>>
>> I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching to
>> {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
>>
>> I have also captured few JMX metric graphs as follows:
>>
>> ​
>> Garbage collection metrics did not look very interesting, hence I have
>> not added the same.
>>
>> Any ideas what more I can do to debug the system and increase its read
>> throughput?
>>
>> Thanks
>> SG
>>
>>
>>
>> On Sat, Mar 11, 2017 at 2:33 PM, Jeff Jirsa <jj...@gmail.com> wrote:
>>
>> 5.5G of ram isn't very much for a jvm based database. Having fewer
>> instances with more ram will likely give you better performance.
>>
>> Also, this is probably a better question for the user list than the Dev
>> list
>>
>> --
>> Jeff Jirsa
>>
>>
>> > On Mar 11, 2017, at 1:20 PM, S G <sg...@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > I have a 9 node cassandra cluster, each node has:
>> > RAM: 5.5gb
>> > Disk: 64gb
>> > C* version: 3.3
>> > Java: 1.8.0_51
>> >
>> > The cluster stores about 2 million rows and partition keys for all these
>> > rows are unique.
>> >
>> > I am stress-reading it from a 12-node client cluster.
>> > Each read-client has 50 threads, so total 600 read-threads from 12
>> machines.
>> > Each read query is get-by-primary-key where keys are read randomly from
>> a
>> > file having all primary-keys.
>> >
>> >
>> > I am able to get only 15,000 reads/second from the entire system.
>> > Is this the best read performance I can expect?
>> > Are there any benchmarks I can compare against?
>> >
>> >
>> > I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching
>> to
>> > {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
>> >
>> > Thx,
>> > SG
>>
>>
>>
>>

Re: How can I scale my read rate?

[Arvydas Jonusonis <ar...@gmail.com>]

It would be interesting to see some of the driver metrics (in your stress
test tool) - if you enable JMX, they should be exposed by default.

Also, are you using session.execute(..) or session.executeAsync(..) ?

Arvydas

On Thu, Mar 16, 2017 at 00:44 Eric Stevens <mi...@gmail.com> wrote:

> I assume your graphs are from a node on the Cassandra cluster.  You're
> really not stressing out that cluster, load average under 1 is an extremely
> bored cluster wishing you would give it something to work on.  I have a
> hunch that the limiting factor might be in your read stress testing tool.
>
> Are you doing sequential reads where each thread waits for the previous
> read to complete before attempting the next read?  You might be spending
> most of your time just waiting for network round trips.  I would suggest
> slowly ramping up the number of simultaneous outstanding reads per thread
> until you start to get a non-trivial amount of read timeout exceptions - at
> that point is where you've hit your maximum read throughput.  Depending on
> what kind of disks you have there you might expect per-node load averages
> to be in the 30+ range (maybe lower, that heap size is a bit tight).
>
> On Wed, Mar 15, 2017 at 4:22 PM Jonathan Haddad <jo...@jonhaddad.com> wrote:
>
> Really no need to be rude.  Why the hostility?
>
> On Wed, Mar 15, 2017 at 2:12 PM daemeon reiydelle <da...@gmail.com>
> wrote:
>
> with that brain damaged memory setting yes.
>
>
> *.......*
>
>
>
> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 <(415)%20501-0198>London
> (+44) (0) 20 8144 9872 <+44%2020%208144%209872>*
>
> On Tue, Mar 14, 2017 at 3:45 PM, S G <sg...@gmail.com> wrote:
>
> Hi,
>
> I am trying to scale my read throughput.
>
> I have a 12 node Cassandra cluster, each node has:
> RAM: 5.5gb
> Disk: 64gb
> C* version: 3.3
> Java: 1.8.0_51
>
> The cluster has 2 tables - each with 2 million rows.
> Partition keys for all these rows are unique.
>
> I am stress-reading it from a 15-node client cluster.
> Each read-client has 40 threads, so total 600 read-threads from 12
> machines.
> Each read query is get-by-primary-key where keys are read randomly from a
> file having all primary-keys.
>
>
> I am able to get only 15,000 reads/second from the entire system.
> Is this the best read performance I can expect?
> Are there any benchmarks I can compare against?
>
>
> I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching to
> {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
>
> I have also captured few JMX metric graphs as follows:
>
> ​
> Garbage collection metrics did not look very interesting, hence I have not
> added the same.
>
> Any ideas what more I can do to debug the system and increase its read
> throughput?
>
> Thanks
> SG
>
>
>
> On Sat, Mar 11, 2017 at 2:33 PM, Jeff Jirsa <jj...@gmail.com> wrote:
>
> 5.5G of ram isn't very much for a jvm based database. Having fewer
> instances with more ram will likely give you better performance.
>
> Also, this is probably a better question for the user list than the Dev
> list
>
> --
> Jeff Jirsa
>
>
> > On Mar 11, 2017, at 1:20 PM, S G <sg...@gmail.com> wrote:
> >
> > Hi,
> >
> > I have a 9 node cassandra cluster, each node has:
> > RAM: 5.5gb
> > Disk: 64gb
> > C* version: 3.3
> > Java: 1.8.0_51
> >
> > The cluster stores about 2 million rows and partition keys for all these
> > rows are unique.
> >
> > I am stress-reading it from a 12-node client cluster.
> > Each read-client has 50 threads, so total 600 read-threads from 12
> machines.
> > Each read query is get-by-primary-key where keys are read randomly from a
> > file having all primary-keys.
> >
> >
> > I am able to get only 15,000 reads/second from the entire system.
> > Is this the best read performance I can expect?
> > Are there any benchmarks I can compare against?
> >
> >
> > I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching to
> > {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
> >
> > Thx,
> > SG
>
>
>
>

Re: How can I scale my read rate?

[Eric Stevens <mi...@gmail.com>]

I assume your graphs are from a node on the Cassandra cluster.  You're
really not stressing out that cluster, load average under 1 is an extremely
bored cluster wishing you would give it something to work on.  I have a
hunch that the limiting factor might be in your read stress testing tool.

Are you doing sequential reads where each thread waits for the previous
read to complete before attempting the next read?  You might be spending
most of your time just waiting for network round trips.  I would suggest
slowly ramping up the number of simultaneous outstanding reads per thread
until you start to get a non-trivial amount of read timeout exceptions - at
that point is where you've hit your maximum read throughput.  Depending on
what kind of disks you have there you might expect per-node load averages
to be in the 30+ range (maybe lower, that heap size is a bit tight).

On Wed, Mar 15, 2017 at 4:22 PM Jonathan Haddad <jo...@jonhaddad.com> wrote:

> Really no need to be rude.  Why the hostility?
>
> On Wed, Mar 15, 2017 at 2:12 PM daemeon reiydelle <da...@gmail.com>
> wrote:
>
> with that brain damaged memory setting yes.
>
>
> *.......*
>
>
>
> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 <(415)%20501-0198>London
> (+44) (0) 20 8144 9872 <+44%2020%208144%209872>*
>
> On Tue, Mar 14, 2017 at 3:45 PM, S G <sg...@gmail.com> wrote:
>
> Hi,
>
> I am trying to scale my read throughput.
>
> I have a 12 node Cassandra cluster, each node has:
> RAM: 5.5gb
> Disk: 64gb
> C* version: 3.3
> Java: 1.8.0_51
>
> The cluster has 2 tables - each with 2 million rows.
> Partition keys for all these rows are unique.
>
> I am stress-reading it from a 15-node client cluster.
> Each read-client has 40 threads, so total 600 read-threads from 12
> machines.
> Each read query is get-by-primary-key where keys are read randomly from a
> file having all primary-keys.
>
>
> I am able to get only 15,000 reads/second from the entire system.
> Is this the best read performance I can expect?
> Are there any benchmarks I can compare against?
>
>
> I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching to
> {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
>
> I have also captured few JMX metric graphs as follows:
>
> ​
> Garbage collection metrics did not look very interesting, hence I have not
> added the same.
>
> Any ideas what more I can do to debug the system and increase its read
> throughput?
>
> Thanks
> SG
>
>
>
> On Sat, Mar 11, 2017 at 2:33 PM, Jeff Jirsa <jj...@gmail.com> wrote:
>
> 5.5G of ram isn't very much for a jvm based database. Having fewer
> instances with more ram will likely give you better performance.
>
> Also, this is probably a better question for the user list than the Dev
> list
>
> --
> Jeff Jirsa
>
>
> > On Mar 11, 2017, at 1:20 PM, S G <sg...@gmail.com> wrote:
> >
> > Hi,
> >
> > I have a 9 node cassandra cluster, each node has:
> > RAM: 5.5gb
> > Disk: 64gb
> > C* version: 3.3
> > Java: 1.8.0_51
> >
> > The cluster stores about 2 million rows and partition keys for all these
> > rows are unique.
> >
> > I am stress-reading it from a 12-node client cluster.
> > Each read-client has 50 threads, so total 600 read-threads from 12
> machines.
> > Each read query is get-by-primary-key where keys are read randomly from a
> > file having all primary-keys.
> >
> >
> > I am able to get only 15,000 reads/second from the entire system.
> > Is this the best read performance I can expect?
> > Are there any benchmarks I can compare against?
> >
> >
> > I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching to
> > {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
> >
> > Thx,
> > SG
>
>
>
>

Re: How can I scale my read rate?

[Jonathan Haddad <jo...@jonhaddad.com>]

Really no need to be rude.  Why the hostility?

On Wed, Mar 15, 2017 at 2:12 PM daemeon reiydelle <da...@gmail.com>
wrote:

> with that brain damaged memory setting yes.
>
>
> *.......*
>
>
>
> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 <(415)%20501-0198>London
> (+44) (0) 20 8144 9872 <+44%2020%208144%209872>*
>
> On Tue, Mar 14, 2017 at 3:45 PM, S G <sg...@gmail.com> wrote:
>
> Hi,
>
> I am trying to scale my read throughput.
>
> I have a 12 node Cassandra cluster, each node has:
> RAM: 5.5gb
> Disk: 64gb
> C* version: 3.3
> Java: 1.8.0_51
>
> The cluster has 2 tables - each with 2 million rows.
> Partition keys for all these rows are unique.
>
> I am stress-reading it from a 15-node client cluster.
> Each read-client has 40 threads, so total 600 read-threads from 12
> machines.
> Each read query is get-by-primary-key where keys are read randomly from a
> file having all primary-keys.
>
>
> I am able to get only 15,000 reads/second from the entire system.
> Is this the best read performance I can expect?
> Are there any benchmarks I can compare against?
>
>
> I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching to
> {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
>
> I have also captured few JMX metric graphs as follows:
>
> ​
> Garbage collection metrics did not look very interesting, hence I have not
> added the same.
>
> Any ideas what more I can do to debug the system and increase its read
> throughput?
>
> Thanks
> SG
>
>
>
> On Sat, Mar 11, 2017 at 2:33 PM, Jeff Jirsa <jj...@gmail.com> wrote:
>
> 5.5G of ram isn't very much for a jvm based database. Having fewer
> instances with more ram will likely give you better performance.
>
> Also, this is probably a better question for the user list than the Dev
> list
>
> --
> Jeff Jirsa
>
>
> > On Mar 11, 2017, at 1:20 PM, S G <sg...@gmail.com> wrote:
> >
> > Hi,
> >
> > I have a 9 node cassandra cluster, each node has:
> > RAM: 5.5gb
> > Disk: 64gb
> > C* version: 3.3
> > Java: 1.8.0_51
> >
> > The cluster stores about 2 million rows and partition keys for all these
> > rows are unique.
> >
> > I am stress-reading it from a 12-node client cluster.
> > Each read-client has 50 threads, so total 600 read-threads from 12
> machines.
> > Each read query is get-by-primary-key where keys are read randomly from a
> > file having all primary-keys.
> >
> >
> > I am able to get only 15,000 reads/second from the entire system.
> > Is this the best read performance I can expect?
> > Are there any benchmarks I can compare against?
> >
> >
> > I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching to
> > {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
> >
> > Thx,
> > SG
>
>
>
>

Re: How can I scale my read rate?

[daemeon reiydelle <da...@gmail.com>]

with that brain damaged memory setting yes.


*.......*



*Daemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872*

On Tue, Mar 14, 2017 at 3:45 PM, S G <sg...@gmail.com> wrote:

> Hi,
>
> I am trying to scale my read throughput.
>
> I have a 12 node Cassandra cluster, each node has:
> RAM: 5.5gb
> Disk: 64gb
> C* version: 3.3
> Java: 1.8.0_51
>
> The cluster has 2 tables - each with 2 million rows.
> Partition keys for all these rows are unique.
>
> I am stress-reading it from a 15-node client cluster.
> Each read-client has 40 threads, so total 600 read-threads from 12
> machines.
> Each read query is get-by-primary-key where keys are read randomly from a
> file having all primary-keys.
>
>
> I am able to get only 15,000 reads/second from the entire system.
> Is this the best read performance I can expect?
> Are there any benchmarks I can compare against?
>
>
> I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching to
> {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
>
> I have also captured few JMX metric graphs as follows:
>
> ​
> Garbage collection metrics did not look very interesting, hence I have not
> added the same.
>
> Any ideas what more I can do to debug the system and increase its read
> throughput?
>
> Thanks
> SG
>
>
>
> On Sat, Mar 11, 2017 at 2:33 PM, Jeff Jirsa <jj...@gmail.com> wrote:
>
>> 5.5G of ram isn't very much for a jvm based database. Having fewer
>> instances with more ram will likely give you better performance.
>>
>> Also, this is probably a better question for the user list than the Dev
>> list
>>
>> --
>> Jeff Jirsa
>>
>>
>> > On Mar 11, 2017, at 1:20 PM, S G <sg...@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > I have a 9 node cassandra cluster, each node has:
>> > RAM: 5.5gb
>> > Disk: 64gb
>> > C* version: 3.3
>> > Java: 1.8.0_51
>> >
>> > The cluster stores about 2 million rows and partition keys for all these
>> > rows are unique.
>> >
>> > I am stress-reading it from a 12-node client cluster.
>> > Each read-client has 50 threads, so total 600 read-threads from 12
>> machines.
>> > Each read query is get-by-primary-key where keys are read randomly from
>> a
>> > file having all primary-keys.
>> >
>> >
>> > I am able to get only 15,000 reads/second from the entire system.
>> > Is this the best read performance I can expect?
>> > Are there any benchmarks I can compare against?
>> >
>> >
>> > I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching
>> to
>> > {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
>> >
>> > Thx,
>> > SG
>>
>
>

Re: How can I scale my read rate?

[S G <sg...@gmail.com>]

Hi,

I am trying to scale my read throughput.

I have a 12 node Cassandra cluster, each node has:
RAM: 5.5gb
Disk: 64gb
C* version: 3.3
Java: 1.8.0_51

The cluster has 2 tables - each with 2 million rows.
Partition keys for all these rows are unique.

I am stress-reading it from a 15-node client cluster.
Each read-client has 40 threads, so total 600 read-threads from 12 machines.
Each read query is get-by-primary-key where keys are read randomly from a
file having all primary-keys.


I am able to get only 15,000 reads/second from the entire system.
Is this the best read performance I can expect?
Are there any benchmarks I can compare against?


I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching to
{'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.

I have also captured few JMX metric graphs as follows:

​
Garbage collection metrics did not look very interesting, hence I have not
added the same.

Any ideas what more I can do to debug the system and increase its read
throughput?

Thanks
SG



On Sat, Mar 11, 2017 at 2:33 PM, Jeff Jirsa <jj...@gmail.com> wrote:

> 5.5G of ram isn't very much for a jvm based database. Having fewer
> instances with more ram will likely give you better performance.
>
> Also, this is probably a better question for the user list than the Dev
> list
>
> --
> Jeff Jirsa
>
>
> > On Mar 11, 2017, at 1:20 PM, S G <sg...@gmail.com> wrote:
> >
> > Hi,
> >
> > I have a 9 node cassandra cluster, each node has:
> > RAM: 5.5gb
> > Disk: 64gb
> > C* version: 3.3
> > Java: 1.8.0_51
> >
> > The cluster stores about 2 million rows and partition keys for all these
> > rows are unique.
> >
> > I am stress-reading it from a 12-node client cluster.
> > Each read-client has 50 threads, so total 600 read-threads from 12
> machines.
> > Each read query is get-by-primary-key where keys are read randomly from a
> > file having all primary-keys.
> >
> >
> > I am able to get only 15,000 reads/second from the entire system.
> > Is this the best read performance I can expect?
> > Are there any benchmarks I can compare against?
> >
> >
> > I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching to
> > {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
> >
> > Thx,
> > SG
>

Re: How can I scale my read rate?

[Jeff Jirsa <jj...@gmail.com>]

5.5G of ram isn't very much for a jvm based database. Having fewer instances with more ram will likely give you better performance.

Also, this is probably a better question for the user list than the Dev list 

-- 
Jeff Jirsa


> On Mar 11, 2017, at 1:20 PM, S G <sg...@gmail.com> wrote:
> 
> Hi,
> 
> I have a 9 node cassandra cluster, each node has:
> RAM: 5.5gb
> Disk: 64gb
> C* version: 3.3
> Java: 1.8.0_51
> 
> The cluster stores about 2 million rows and partition keys for all these
> rows are unique.
> 
> I am stress-reading it from a 12-node client cluster.
> Each read-client has 50 threads, so total 600 read-threads from 12 machines.
> Each read query is get-by-primary-key where keys are read randomly from a
> file having all primary-keys.
> 
> 
> I am able to get only 15,000 reads/second from the entire system.
> Is this the best read performance I can expect?
> Are there any benchmarks I can compare against?
> 
> 
> I tried setting bloom_filter_fp_chance from 0.01 to 0.0001 and caching to
> {'keys': 'ALL', 'rows_per_partition': 'ALL'} but it had no effect.
> 
> Thx,
> SG