You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by onmstester onmstester <on...@zoho.com> on 2018/02/18 11:29:31 UTC
Cassandra cluster: could not reach linear scalability
I've configured a simple cluster using two PC with identical spec:
cpu core i5 RAM: 8GB ddr3 Disk: 1TB 5400rpm Network: 1 G (I've test it with iperf, it really is!)
using the common configs described in many sites including datastax itself:
cluster_name: 'MyCassandraCluster' num_tokens: 256 seed_provider: - class_name: org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: "192.168.1.1,192.168.1.2" listen_address: rpc_address: 0.0.0.0 endpoint_snitch: GossipingPropertyFileSnitch
Running stress tool:
cassandra-stress write n=1000000 -rate threads=1000 -mode native cql3 -node 192.168.1.1,192.168.1.2
Over each node it shows 39 K writes/seconds, but running the same stress tool command on cluster of both nodes shows 45 K writes/seconds. I've done all the tuning mentioned by apache and datastax. There are many use cases on the net proving Cassandra linear Scalability So what is wrong with my cluster?
Sent using Zoho Mail
Re: Cassandra cluster: could not reach linear scalability
Posted by Rahul Singh <ra...@gmail.com>.
If that is the case you could also try to run more stress from another machine as well.
--
Rahul Singh
rahul.singh@anant.us
Anant Corporation
On Feb 18, 2018, 2:37 PM -0500, Jeff Jirsa <jj...@gmail.com>, wrote:
> Stress client may be cpu bound as well
>
> --
> Jeff Jirsa
>
>
> On Feb 18, 2018, at 7:40 AM, onmstester onmstester <on...@zoho.com> wrote:
>
> > I'm running tests on separate machine (not member of the cluster)
> > I'm using the default data model of cassandra-stress tool : keyspace1 and table: standard1. nothing special on network or data traffic. Network capable of 1 G and tested it with ipperf.
> > iftop shows maximum of 48Mbit traffic between nodes in cluster.
> > Have not seen any warning in log files.
> > I'm monitoring cassandra during runtime using jvisual vm and never saw any GC chokepoints, cpu is below 40% always. I just cant understand why cassandra is limmiting the throughput?!
> > using top, fps and write per seconds are not showing any problems
> >
> > Sent using Zoho Mail
> >
> >
> > ---- On Sun, 18 Feb 2018 18:42:48 +0330 Rahul Singh <ra...@gmail.com> wrote ----
> >
> > > Got it.
> > >
> > > Here are some other questions.
> > >
> > > Are you running the test on separate machine or one of the cluster members?
> > >
> > > When configuring Cassandra stress what data model did you end up using? ( do you see Data or traffic skew?)
> > >
> > > Do you see any wide partitions or Tombstone warnings on either node.
> > >
> > > Have you visualized the GC logs using something like VisualVM or hubspots GC visualizer? This is to see if there are chokepoints in the GC cycle.
> > >
> > >
> > > --
> > > Rahul Singh
> > > rahul.singh@anant.us
> > >
> > > Anant Corporation
> > >
> > > On Feb 18, 2018, 9:23 AM -0500, onmstester onmstester <on...@zoho.com>, wrote:
> > >
> > > > But monitoring cassandra with jmx using jvisualVM shows no problem, less than 30% of heap size used
> > > >
> > > > Sent using Zoho Mail
> > > >
> > > >
> > > > ---- On Sun, 18 Feb 2018 17:26:59 +0330 Rahul Singh <ra...@gmail.com> wrote ----
> > > >
> > > > > You don’t don’t have enough memory. That’s just a start.
> > > > >
> > > > > --
> > > > > Rahul Singh
> > > > > rahul.singh@anant.us
> > > > >
> > > > > Anant Corporation
> > > > >
> > > > > On Feb 18, 2018, 6:29 AM -0500, onmstester onmstester <on...@zoho.com>, wrote:
> > > > >
> > > > > > I've configured a simple cluster using two PC with identical spec:
> > > > > > cpu core i5
> > > > > > RAM: 8GB ddr3
> > > > > > Disk: 1TB 5400rpm
> > > > > > Network: 1 G (I've test it with iperf, it really is!)
> > > > > >
> > > > > > using the common configs described in many sites including datastax itself:
> > > > > > cluster_name: 'MyCassandraCluster'
> > > > > > num_tokens: 256
> > > > > > seed_provider:
> > > > > > - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> > > > > > parameters:
> > > > > > - seeds: "192.168.1.1,192.168.1.2"
> > > > > > listen_address:
> > > > > > rpc_address: 0.0.0.0
> > > > > > endpoint_snitch: GossipingPropertyFileSnitch
> > > > > >
> > > > > > Running stress tool:
> > > > > > cassandra-stress write n=1000000 -rate threads=1000 -mode native cql3 -node 192.168.1.1,192.168.1.2
> > > > > >
> > > > > > Over each node it shows 39 K writes/seconds, but running the same stress tool command on cluster of both nodes shows 45 K writes/seconds. I've done all the tuning mentioned by apache and datastax. There are many use cases on the net proving Cassandra linear Scalability So what is wrong with my cluster?
> > > > > >
> > > > > > Sent using Zoho Mail
> > > > > >
> > > >
> >
> >
Re: Cassandra cluster: could not reach linear scalability
Posted by Jeff Jirsa <jj...@gmail.com>.
Stress client may be cpu bound as well
--
Jeff Jirsa
> On Feb 18, 2018, at 7:40 AM, onmstester onmstester <on...@zoho.com> wrote:
>
> I'm running tests on separate machine (not member of the cluster)
> I'm using the default data model of cassandra-stress tool : keyspace1 and table: standard1. nothing special on network or data traffic. Network capable of 1 G and tested it with ipperf.
> iftop shows maximum of 48Mbit traffic between nodes in cluster.
> Have not seen any warning in log files.
> I'm monitoring cassandra during runtime using jvisual vm and never saw any GC chokepoints, cpu is below 40% always. I just cant understand why cassandra is limmiting the throughput?!
> using top, fps and write per seconds are not showing any problems
>
> Sent using Zoho Mail
>
>
>
> ---- On Sun, 18 Feb 2018 18:42:48 +0330 Rahul Singh <ra...@gmail.com> wrote ----
>
> Got it.
>
> Here are some other questions.
>
> Are you running the test on separate machine or one of the cluster members?
>
> When configuring Cassandra stress what data model did you end up using? ( do you see Data or traffic skew?)
>
> Do you see any wide partitions or Tombstone warnings on either node.
>
> Have you visualized the GC logs using something like VisualVM or hubspots GC visualizer? This is to see if there are chokepoints in the GC cycle.
>
>
> --
> Rahul Singh
> rahul.singh@anant.us
>
> Anant Corporation
>
> On Feb 18, 2018, 9:23 AM -0500, onmstester onmstester <on...@zoho.com>, wrote:
>
> But monitoring cassandra with jmx using jvisualVM shows no problem, less than 30% of heap size used
>
> Sent using Zoho Mail
>
>
>
> ---- On Sun, 18 Feb 2018 17:26:59 +0330 Rahul Singh <ra...@gmail.com> wrote ----
>
> You don’t don’t have enough memory. That’s just a start.
>
> --
> Rahul Singh
> rahul.singh@anant.us
>
> Anant Corporation
>
> On Feb 18, 2018, 6:29 AM -0500, onmstester onmstester <on...@zoho.com>, wrote:
>
> I've configured a simple cluster using two PC with identical spec:
>
> cpu core i5
> RAM: 8GB ddr3
> Disk: 1TB 5400rpm
> Network: 1 G (I've test it with iperf, it really is!)
>
> using the common configs described in many sites including datastax itself:
>
> cluster_name: 'MyCassandraCluster'
> num_tokens: 256
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "192.168.1.1,192.168.1.2"
> listen_address:
> rpc_address: 0.0.0.0
> endpoint_snitch: GossipingPropertyFileSnitch
>
> Running stress tool:
>
> cassandra-stress write n=1000000 -rate threads=1000 -mode native cql3 -node 192.168.1.1,192.168.1.2
>
> Over each node it shows 39 K writes/seconds, but running the same stress tool command on cluster of both nodes shows 45 K writes/seconds. I've done all the tuning mentioned by apache and datastax. There are many use cases on the net proving Cassandra linear Scalability So what is wrong with my cluster?
>
>
> Sent using Zoho Mail
>
>
>
>
>
Re: Cassandra cluster: could not reach linear scalability
Posted by onmstester onmstester <on...@zoho.com>.
I'm running tests on separate machine (not member of the cluster)
I'm using the default data model of cassandra-stress tool : keyspace1 and table: standard1. nothing special on network or data traffic. Network capable of 1 G and tested it with ipperf.
iftop shows maximum of 48Mbit traffic between nodes in cluster.
Have not seen any warning in log files.
I'm monitoring cassandra during runtime using jvisual vm and never saw any GC chokepoints, cpu is below 40% always. I just cant understand why cassandra is limmiting the throughput?!
using top, fps and write per seconds are not showing any problems
Sent using Zoho Mail
---- On Sun, 18 Feb 2018 18:42:48 +0330 Rahul Singh <rahul.xavier.singh@gmail.com> wrote ----
Got it.
Here are some other questions.
Are you running the test on separate machine or one of the cluster members?
When configuring Cassandra stress what data model did you end up using? ( do you see Data or traffic skew?)
Do you see any wide partitions or Tombstone warnings on either node.
Have you visualized the GC logs using something like VisualVM or hubspots GC visualizer? This is to see if there are chokepoints in the GC cycle.
--
Rahul Singh
rahul.singh@anant.us
Anant Corporation
On Feb 18, 2018, 9:23 AM -0500, onmstester onmstester <onmstester@zoho.com>, wrote:
But monitoring cassandra with jmx using jvisualVM shows no problem, less than 30% of heap size used
Sent using Zoho Mail
---- On Sun, 18 Feb 2018 17:26:59 +0330 Rahul Singh <rahul.xavier.singh@gmail.com> wrote ----
You don’t don’t have enough memory. That’s just a start.
--
Rahul Singh
rahul.singh@anant.us
Anant Corporation
On Feb 18, 2018, 6:29 AM -0500, onmstester onmstester <onmstester@zoho.com>, wrote:
I've configured a simple cluster using two PC with identical spec:
cpu core i5 RAM: 8GB ddr3 Disk: 1TB 5400rpm Network: 1 G (I've test it with iperf, it really is!)
using the common configs described in many sites including datastax itself:
cluster_name: 'MyCassandraCluster' num_tokens: 256 seed_provider: - class_name: org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: "192.168.1.1,192.168.1.2" listen_address: rpc_address: 0.0.0.0 endpoint_snitch: GossipingPropertyFileSnitch
Running stress tool:
cassandra-stress write n=1000000 -rate threads=1000 -mode native cql3 -node 192.168.1.1,192.168.1.2
Over each node it shows 39 K writes/seconds, but running the same stress tool command on cluster of both nodes shows 45 K writes/seconds. I've done all the tuning mentioned by apache and datastax. There are many use cases on the net proving Cassandra linear Scalability So what is wrong with my cluster?
Sent using Zoho Mail
Re: Cassandra cluster: could not reach linear scalability
Posted by Rahul Singh <ra...@gmail.com>.
Got it.
Here are some other questions.
Are you running the test on separate machine or one of the cluster members?
When configuring Cassandra stress what data model did you end up using? ( do you see Data or traffic skew?)
Do you see any wide partitions or Tombstone warnings on either node.
Have you visualized the GC logs using something like VisualVM or hubspots GC visualizer? This is to see if there are chokepoints in the GC cycle.
--
Rahul Singh
rahul.singh@anant.us
Anant Corporation
On Feb 18, 2018, 9:23 AM -0500, onmstester onmstester <on...@zoho.com>, wrote:
> But monitoring cassandra with jmx using jvisualVM shows no problem, less than 30% of heap size used
>
> Sent using Zoho Mail
>
>
> ---- On Sun, 18 Feb 2018 17:26:59 +0330 Rahul Singh <ra...@gmail.com> wrote ----
>
> > You don’t don’t have enough memory. That’s just a start.
> >
> > --
> > Rahul Singh
> > rahul.singh@anant.us
> >
> > Anant Corporation
> >
> > On Feb 18, 2018, 6:29 AM -0500, onmstester onmstester <on...@zoho.com>, wrote:
> >
> > > I've configured a simple cluster using two PC with identical spec:
> > > cpu core i5
> > > RAM: 8GB ddr3
> > > Disk: 1TB 5400rpm
> > > Network: 1 G (I've test it with iperf, it really is!)
> > >
> > > using the common configs described in many sites including datastax itself:
> > > cluster_name: 'MyCassandraCluster'
> > > num_tokens: 256
> > > seed_provider:
> > > - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> > > parameters:
> > > - seeds: "192.168.1.1,192.168.1.2"
> > > listen_address:
> > > rpc_address: 0.0.0.0
> > > endpoint_snitch: GossipingPropertyFileSnitch
> > >
> > > Running stress tool:
> > > cassandra-stress write n=1000000 -rate threads=1000 -mode native cql3 -node 192.168.1.1,192.168.1.2
> > >
> > > Over each node it shows 39 K writes/seconds, but running the same stress tool command on cluster of both nodes shows 45 K writes/seconds. I've done all the tuning mentioned by apache and datastax. There are many use cases on the net proving Cassandra linear Scalability So what is wrong with my cluster?
> > >
> > > Sent using Zoho Mail
> > >
>
>
Re: Cassandra cluster: could not reach linear scalability
Posted by onmstester onmstester <on...@zoho.com>.
But monitoring cassandra with jmx using jvisualVM shows no problem, less than 30% of heap size used
Sent using Zoho Mail
---- On Sun, 18 Feb 2018 17:26:59 +0330 Rahul Singh <rahul.xavier.singh@gmail.com> wrote ----
You don’t don’t have enough memory. That’s just a start.
--
Rahul Singh
rahul.singh@anant.us
Anant Corporation
On Feb 18, 2018, 6:29 AM -0500, onmstester onmstester <onmstester@zoho.com>, wrote:
I've configured a simple cluster using two PC with identical spec:
cpu core i5 RAM: 8GB ddr3 Disk: 1TB 5400rpm Network: 1 G (I've test it with iperf, it really is!)
using the common configs described in many sites including datastax itself:
cluster_name: 'MyCassandraCluster' num_tokens: 256 seed_provider: - class_name: org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: "192.168.1.1,192.168.1.2" listen_address: rpc_address: 0.0.0.0 endpoint_snitch: GossipingPropertyFileSnitch
Running stress tool:
cassandra-stress write n=1000000 -rate threads=1000 -mode native cql3 -node 192.168.1.1,192.168.1.2
Over each node it shows 39 K writes/seconds, but running the same stress tool command on cluster of both nodes shows 45 K writes/seconds. I've done all the tuning mentioned by apache and datastax. There are many use cases on the net proving Cassandra linear Scalability So what is wrong with my cluster?
Sent using Zoho Mail
Re: Cassandra cluster: could not reach linear scalability
Posted by Rahul Singh <ra...@gmail.com>.
You don’t don’t have enough memory. That’s just a start.
--
Rahul Singh
rahul.singh@anant.us
Anant Corporation
On Feb 18, 2018, 6:29 AM -0500, onmstester onmstester <on...@zoho.com>, wrote:
> I've configured a simple cluster using two PC with identical spec:
> cpu core i5
> RAM: 8GB ddr3
> Disk: 1TB 5400rpm
> Network: 1 G (I've test it with iperf, it really is!)
>
> using the common configs described in many sites including datastax itself:
> cluster_name: 'MyCassandraCluster'
> num_tokens: 256
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "192.168.1.1,192.168.1.2"
> listen_address:
> rpc_address: 0.0.0.0
> endpoint_snitch: GossipingPropertyFileSnitch
>
> Running stress tool:
> cassandra-stress write n=1000000 -rate threads=1000 -mode native cql3 -node 192.168.1.1,192.168.1.2
>
> Over each node it shows 39 K writes/seconds, but running the same stress tool command on cluster of both nodes shows 45 K writes/seconds. I've done all the tuning mentioned by apache and datastax. There are many use cases on the net proving Cassandra linear Scalability So what is wrong with my cluster?
>
> Sent using Zoho Mail
>
>
Re: Cassandra cluster: could not reach linear scalability
Posted by onmstester onmstester <on...@zoho.com>.
Thanks for your help,
I'been biased with Cassandra server and forget about the client completely!
Sent using Zoho Mail
---- On Mon, 19 Feb 2018 15:21:03 +0330 Lucas Benevides <lucas@maurobenevides.com.br> wrote ----
Why did you set the number of 1000 threads?
Does it show to be the more performatic when threads = auto?
I have used stress tool in a larger test bed (10 nodes) and my optimal setup was 24 threads.
To check this you must monitor the stress node, both the CPU and I/O. And give it a try with fewer threads.
Lucas Benevides
Ipea
2018-02-18 8:29 GMT-03:00 onmstester onmstester <onmstester@zoho.com>:
I've configured a simple cluster using two PC with identical spec:
cpu core i5 RAM: 8GB ddr3 Disk: 1TB 5400rpm Network: 1 G (I've test it with iperf, it really is!)
using the common configs described in many sites including datastax itself:
cluster_name: 'MyCassandraCluster' num_tokens: 256 seed_provider: - class_name: org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: "192.168.1.1,192.168.1.2" listen_address: rpc_address: 0.0.0.0 endpoint_snitch: GossipingPropertyFileSnitch
Running stress tool:
cassandra-stress write n=1000000 -rate threads=1000 -mode native cql3 -node 192.168.1.1,192.168.1.2
Over each node it shows 39 K writes/seconds, but running the same stress tool command on cluster of both nodes shows 45 K writes/seconds. I've done all the tuning mentioned by apache and datastax. There are many use cases on the net proving Cassandra linear Scalability So what is wrong with my cluster?
Sent using Zoho Mail
Re: Cassandra cluster: could not reach linear scalability
Posted by Lucas Benevides <lu...@maurobenevides.com.br>.
Why did you set the number of 1000 threads?
Does it show to be the more performatic when threads = auto?
I have used stress tool in a larger test bed (10 nodes) and my optimal
setup was 24 threads.
To check this you must monitor the stress node, both the CPU and I/O. And
give it a try with fewer threads.
Lucas Benevides
Ipea
2018-02-18 8:29 GMT-03:00 onmstester onmstester <on...@zoho.com>:
> I've configured a simple cluster using two PC with identical spec:
>
> cpu core i5
> RAM: 8GB ddr3
> Disk: 1TB 5400rpm
> Network: 1 G (I've test it with iperf, it really is!)
>
> using the common configs described in many sites including datastax itself:
>
> cluster_name: 'MyCassandraCluster'
> num_tokens: 256
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "192.168.1.1,192.168.1.2"
> listen_address:
> rpc_address: 0.0.0.0
> endpoint_snitch: GossipingPropertyFileSnitch
>
> Running stress tool:
>
> cassandra-stress write n=1000000 -rate threads=1000 -mode native cql3 -node 192.168.1.1,192.168.1.2
>
> Over each node it shows 39 K writes/seconds, but running the same stress
> tool command on cluster of both nodes shows 45 K writes/seconds. I've done
> all the tuning mentioned by apache and datastax. There are many use cases
> on the net proving Cassandra linear Scalability So what is wrong with my
> cluster?
>
> Sent using Zoho Mail <https://www.zoho.com/mail/>
>
>
>