You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Akshay Suresh <ak...@unotechsoft.com> on 2017/04/17 15:42:07 UTC
Slow writes and Frequent timeouts
Hi
I have set up a cassandra cluster of 8 nodes.
I am using Apache Cassandra 3.9
While using cassandra-stress tool for load testing, I am getting really
slow writes ( low upto few 10-20 writes per second ) along with frequent
timeouts and out of heap space errors.
Kindly let me know how do I resolve this issue.
--
*Disclaimer :* The contents of this e-mail and attachment(s) thereto are
confidential and intended for the named recipient(s) only. It shall not
attach any liability on the originator or Unotech Software Pvt. Ltd. or its
affiliates. Any views or opinions presented in this email are solely those
of the author and may not necessarily reflect the opinions of Unotech
Software Pvt. Ltd. or its affiliates. Any form of reproduction,
dissemination, copying, disclosure, modification, distribution and / or
publication of this message without the prior written consent of the author
of this e-mail is strictly prohibited. If you have received this email in
error please delete it and notify the sender immediately.
Re: Slow writes and Frequent timeouts
Posted by Akshay Suresh <ak...@unotechsoft.com>.
Hi
Ya sure - The contents of my file foo.yaml file are as follows:
keyspace: benchmark
table: foo
insert:
partitions: fixed(1)
columnspec:
- name: field1
size: fixed(10)
- name: field2
size: fixed(11)
population: fixed(1)
- name: field3
size: fixed(10)
- name: field4
size: fixed(3)
population: fixed(3)
- name: field5
size: fixed(11)
* - name: field6 size: uniform(150..300)*
- name: field7
size: fixed(10)
- name: field8
size: fixed(4)
- name: field9
size: fixed(10)
- name: field10
size: fixed(10)
- name: field11
size: fixed(4)
- name: field12
size: uniform(6..12)
- name: field13
size: fixed(4)
population: fixed(6)
- name: field14
size: fixed(4)
population: fixed(6)
- name: field15
size: uniform(10..12)
population: fixed(6)
- name: field16
size: uniform(10..12)
population: fixed(30)
- name: field17
size: fixed(10)
- name: field18
size: uniform(6..12)
population: fixed(30)
- name: field19
size: uniform(6..12)
population: fixed(30)
- name: field20
size: uniform(10..16)
- name: field21
size: uniform(10..16)
- name: field22
size: uniform(10..16)
- name: field23
size: uniform(10..16)
- name: field24
size: uniform(10..16)
- name: field25
size: uniform(10..16)
- name: field26
size: uniform(10..16)
* - name: field27 size: uniform(1000..1500)*
Now the highlighted fields are of type set<text>
I thought since sets are serialized and inserted into Cassandra, I gave a
higher number assuming that would be the number of total characters in that
column.
I did not realize that a size of 1000 for set<text> would insert 1000
elements into the table. Due to this each inserted record was going up to 5
GB.
And I was getting only about 8-10 inserts per second.
Once I changed the size, I started getting about 10,000 - 12,000 inserts
per second.
On Tue, Apr 25, 2017 at 8:10 PM, ZAIDI, ASAD A <az...@att.com> wrote:
> Would you please share what changes you made that proves helpful for you?
>
>
>
>
>
> *From:* Akshay Suresh [mailto:akshay.suresh@unotechsoft.com]
> *Sent:* Tuesday, April 25, 2017 7:33 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Slow writes and Frequent timeouts
>
>
>
> Hi
>
> It turned out to be an issue in my yaml file - Each record had a size in
> GB. I modified my yaml and I am getting
>
> good results.
>
>
>
> Thanks for the support.
>
> Cheers.
>
>
>
> On Tue, Apr 18, 2017 at 11:45 AM, Akshay Suresh <
> akshay.suresh@unotechsoft.com> wrote:
>
> Hi
>
>
>
> - Hardware Specs
>
> RAM: 14GB
>
> Cores: 8
>
> Disk: 440GB SSD
>
> - I have deployed 7 instances ( Earlier, I mentioned 8 but I had to
> decommission 1 due to low disk space ) on AWS.
>
> - Every node is in the same datacenter.
> .
>
> - I have created a custom schema with 28 columns. Created a simple YAML
> file for the stress tool.
> The command is as follows:
> ./cassandra-stress user profile=foo.yml duration=180m ops\(insert=1\)
> cl=one -rate threads=850
>
> - I have used a heap space of 7GB. I am using the default CMS for GC.
>
> I am clearing all the data and logs. I will try to replicate the issue and
> post the exact metrics.
>
>
>
> Thanks in advance.
>
>
>
>
>
> On Mon, Apr 17, 2017 at 10:28 PM, Jon Haddad <jo...@gmail.com>
> wrote:
>
> What are your hardware specs? Where are you running the cluster? Is
> every node in the same physical datacenter? What command are you using to
> run stress?
>
>
>
> On Apr 17, 2017, at 9:57 AM, Akshay Suresh <ak...@unotechsoft.com>
> wrote:
>
>
>
> Hi
>
> I have not done much. Just created a schema with SimpleStrategy and a
> replication factor of 3.
>
> Then I created a yaml file
>
> Now I am running the cassandra stress tool invoking the yaml file - with
> 10,000 records ( no warmup ) with 10 concurrent threads. I am just running
> writes ( no reads )
>
> When I keep running this script again and again eventually my nodes start
> going down and I start getting timeouts and out of memory errors.
>
>
>
>
>
> On Mon, Apr 17, 2017 at 9:23 PM, Hannu Kröger <hk...@gmail.com> wrote:
>
> It would help to know what kind queries are slow.
>
>
>
> Hannu
>
>
>
> On 17 Apr 2017, at 18:42, Akshay Suresh <ak...@unotechsoft.com>
> wrote:
>
>
>
> Hi
>
> I have set up a cassandra cluster of 8 nodes.
>
> I am using Apache Cassandra 3.9
>
> While using cassandra-stress tool for load testing, I am getting really
> slow writes ( low upto few 10-20 writes per second ) along with frequent
> timeouts and out of heap space errors.
>
> Kindly let me know how do I resolve this issue.
>
>
>
> *Disclaimer :* The contents of this e-mail and attachment(s) thereto are
> confidential and intended for the named recipient(s) only. It shall not
> attach any liability on the originator or Unotech Software Pvt. Ltd. or its
> affiliates. Any views or opinions presented in this email are solely those
> of the author and may not necessarily reflect the opinions of Unotech
> Software Pvt. Ltd. or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification, distribution and / or
> publication of this message without the prior written consent of the author
> of this e-mail is strictly prohibited. If you have received this email in
> error please delete it and notify the sender immediately.
>
>
>
>
>
>
> --
>
> Regards,
>
>
> *Akshay Suresh*
> *Unotech Software Pvt. Ltd*
>
> *
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.unotechsoft.com_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=6oi1htUxdeiGO6J_2mbMkxeZudlH1ZS-t4D0H5zA2VQ&s=4sIYPws_WQ4azv3GShryRfttnaxmcY-22IDBVQsP_C0&e=>*
>
> M
>
> :
>
> +91 99309 80509
>
> O
>
> :
>
> +91 (22) 2687 9402
>
> A
>
> :
>
> D Wing, 7th floor, 32 Corporate Avenue, Off Mahakali Caves Road, Andheri
> (E), Mumbai, 400093
>
> W
>
> :
>
> www.unotechsoft.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.unotechsoft.com_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=6oi1htUxdeiGO6J_2mbMkxeZudlH1ZS-t4D0H5zA2VQ&s=4sIYPws_WQ4azv3GShryRfttnaxmcY-22IDBVQsP_C0&e=>
>
>
> *Disclaimer :* The contents of this e-mail and attachment(s) thereto are
> confidential and intended for the named recipient(s) only. It shall not
> attach any liability on the originator or Unotech Software Pvt. Ltd. or its
> affiliates. Any views or opinions presented in this email are solely those
> of the author and may not necessarily reflect the opinions of Unotech
> Software Pvt. Ltd. or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification, distribution and / or
> publication of this message without the prior written consent of the author
> of this e-mail is strictly prohibited. If you have received this email in
> error please delete it and notify the sender immediately.
>
>
>
>
>
>
> --
>
> Regards,
>
>
> *Akshay Suresh*
> *Unotech Software Pvt. Ltd*
>
> *
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.unotechsoft.com_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=6oi1htUxdeiGO6J_2mbMkxeZudlH1ZS-t4D0H5zA2VQ&s=4sIYPws_WQ4azv3GShryRfttnaxmcY-22IDBVQsP_C0&e=>*
>
> M
>
> :
>
> +91 99309 80509
>
> O
>
> :
>
> +91 (22) 2687 9402
>
> A
>
> :
>
> D Wing, 7th floor, 32 Corporate Avenue, Off Mahakali Caves Road, Andheri
> (E), Mumbai, 400093
>
> W
>
> :
>
> www.unotechsoft.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.unotechsoft.com_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=6oi1htUxdeiGO6J_2mbMkxeZudlH1ZS-t4D0H5zA2VQ&s=4sIYPws_WQ4azv3GShryRfttnaxmcY-22IDBVQsP_C0&e=>
>
>
>
>
> --
>
> Regards,
>
>
> *Akshay Suresh*
> *Unotech Software Pvt. Ltd*
>
> *
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.unotechsoft.com_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=6oi1htUxdeiGO6J_2mbMkxeZudlH1ZS-t4D0H5zA2VQ&s=4sIYPws_WQ4azv3GShryRfttnaxmcY-22IDBVQsP_C0&e=>*
>
> M
>
> :
>
> +91 99309 80509
>
> O
>
> :
>
> +91 (22) 2687 9402
>
> A
>
> :
>
> D Wing, 7th floor, 32 Corporate Avenue, Off Mahakali Caves Road, Andheri
> (E), Mumbai, 400093
>
> W
>
> :
>
> www.unotechsoft.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.unotechsoft.com_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=6oi1htUxdeiGO6J_2mbMkxeZudlH1ZS-t4D0H5zA2VQ&s=4sIYPws_WQ4azv3GShryRfttnaxmcY-22IDBVQsP_C0&e=>
>
>
> *Disclaimer :* The contents of this e-mail and attachment(s) thereto are
> confidential and intended for the named recipient(s) only. It shall not
> attach any liability on the originator or Unotech Software Pvt. Ltd. or its
> affiliates. Any views or opinions presented in this email are solely those
> of the author and may not necessarily reflect the opinions of Unotech
> Software Pvt. Ltd. or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification, distribution and / or
> publication of this message without the prior written consent of the author
> of this e-mail is strictly prohibited. If you have received this email in
> error please delete it and notify the sender immediately.
>
--
Regards,
*Akshay Suresh*
*Unotech Software Pvt. Ltd*
* <http://www.unotechsoft.com/>*
M : +91 99309 80509
O : +91 (22) 2687 9402
A : D Wing, 7th floor, 32 Corporate Avenue, Off Mahakali Caves Road,
Andheri (E), Mumbai, 400093
W : www.unotechsoft.com
--
*Disclaimer :* The contents of this e-mail and attachment(s) thereto are
confidential and intended for the named recipient(s) only. It shall not
attach any liability on the originator or Unotech Software Pvt. Ltd. or its
affiliates. Any views or opinions presented in this email are solely those
of the author and may not necessarily reflect the opinions of Unotech
Software Pvt. Ltd. or its affiliates. Any form of reproduction,
dissemination, copying, disclosure, modification, distribution and / or
publication of this message without the prior written consent of the author
of this e-mail is strictly prohibited. If you have received this email in
error please delete it and notify the sender immediately.
RE: Slow writes and Frequent timeouts
Posted by "ZAIDI, ASAD A" <az...@att.com>.
Would you please share what changes you made that proves helpful for you?
From: Akshay Suresh [mailto:akshay.suresh@unotechsoft.com]
Sent: Tuesday, April 25, 2017 7:33 AM
To: user@cassandra.apache.org
Subject: Re: Slow writes and Frequent timeouts
Hi
It turned out to be an issue in my yaml file - Each record had a size in GB. I modified my yaml and I am getting
good results.
Thanks for the support.
Cheers.
On Tue, Apr 18, 2017 at 11:45 AM, Akshay Suresh <ak...@unotechsoft.com>> wrote:
Hi
- Hardware Specs
RAM: 14GB
Cores: 8
Disk: 440GB SSD
- I have deployed 7 instances ( Earlier, I mentioned 8 but I had to decommission 1 due to low disk space ) on AWS.
- Every node is in the same datacenter.
.
- I have created a custom schema with 28 columns. Created a simple YAML file for the stress tool.
The command is as follows:
./cassandra-stress user profile=foo.yml duration=180m ops\(insert=1\) cl=one -rate threads=850
- I have used a heap space of 7GB. I am using the default CMS for GC.
I am clearing all the data and logs. I will try to replicate the issue and post the exact metrics.
Thanks in advance.
On Mon, Apr 17, 2017 at 10:28 PM, Jon Haddad <jo...@gmail.com>> wrote:
What are your hardware specs? Where are you running the cluster? Is every node in the same physical datacenter? What command are you using to run stress?
On Apr 17, 2017, at 9:57 AM, Akshay Suresh <ak...@unotechsoft.com>> wrote:
Hi
I have not done much. Just created a schema with SimpleStrategy and a replication factor of 3.
Then I created a yaml file
Now I am running the cassandra stress tool invoking the yaml file - with 10,000 records ( no warmup ) with 10 concurrent threads. I am just running writes ( no reads )
When I keep running this script again and again eventually my nodes start going down and I start getting timeouts and out of memory errors.
On Mon, Apr 17, 2017 at 9:23 PM, Hannu Kröger <hk...@gmail.com>> wrote:
It would help to know what kind queries are slow.
Hannu
On 17 Apr 2017, at 18:42, Akshay Suresh <ak...@unotechsoft.com>> wrote:
Hi
I have set up a cassandra cluster of 8 nodes.
I am using Apache Cassandra 3.9
While using cassandra-stress tool for load testing, I am getting really slow writes ( low upto few 10-20 writes per second ) along with frequent timeouts and out of heap space errors.
Kindly let me know how do I resolve this issue.
Disclaimer : The contents of this e-mail and attachment(s) thereto are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or Unotech Software Pvt. Ltd. or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of Unotech Software Pvt. Ltd. or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
--
Regards,
Akshay Suresh
Unotech Software Pvt. Ltd
[https://docs.google.com/uc?export=download&id=0B5gpZneW6JQpUU5Ob3lDQVA0NUk&revid=0B5gpZneW6JQpVVNBKzhhRzBqenc0M2RRdmFZQTdPQUgxUEE4PQ]<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.unotechsoft.com_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=6oi1htUxdeiGO6J_2mbMkxeZudlH1ZS-t4D0H5zA2VQ&s=4sIYPws_WQ4azv3GShryRfttnaxmcY-22IDBVQsP_C0&e=>
M
:
+91 99309 80509
O
:
+91 (22) 2687 9402
A
:
D Wing, 7th floor, 32 Corporate Avenue, Off Mahakali Caves Road, Andheri (E), Mumbai, 400093
W
:
www.unotechsoft.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.unotechsoft.com_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=6oi1htUxdeiGO6J_2mbMkxeZudlH1ZS-t4D0H5zA2VQ&s=4sIYPws_WQ4azv3GShryRfttnaxmcY-22IDBVQsP_C0&e=>
Disclaimer : The contents of this e-mail and attachment(s) thereto are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or Unotech Software Pvt. Ltd. or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of Unotech Software Pvt. Ltd. or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
--
Regards,
Akshay Suresh
Unotech Software Pvt. Ltd
[https://docs.google.com/uc?export=download&id=0B5gpZneW6JQpUU5Ob3lDQVA0NUk&revid=0B5gpZneW6JQpVVNBKzhhRzBqenc0M2RRdmFZQTdPQUgxUEE4PQ]<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.unotechsoft.com_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=6oi1htUxdeiGO6J_2mbMkxeZudlH1ZS-t4D0H5zA2VQ&s=4sIYPws_WQ4azv3GShryRfttnaxmcY-22IDBVQsP_C0&e=>
M
:
+91 99309 80509
O
:
+91 (22) 2687 9402
A
:
D Wing, 7th floor, 32 Corporate Avenue, Off Mahakali Caves Road, Andheri (E), Mumbai, 400093
W
:
www.unotechsoft.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.unotechsoft.com_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=6oi1htUxdeiGO6J_2mbMkxeZudlH1ZS-t4D0H5zA2VQ&s=4sIYPws_WQ4azv3GShryRfttnaxmcY-22IDBVQsP_C0&e=>
--
Regards,
Akshay Suresh
Unotech Software Pvt. Ltd
[https://docs.google.com/uc?export=download&id=0B5gpZneW6JQpUU5Ob3lDQVA0NUk&revid=0B5gpZneW6JQpVVNBKzhhRzBqenc0M2RRdmFZQTdPQUgxUEE4PQ]<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.unotechsoft.com_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=6oi1htUxdeiGO6J_2mbMkxeZudlH1ZS-t4D0H5zA2VQ&s=4sIYPws_WQ4azv3GShryRfttnaxmcY-22IDBVQsP_C0&e=>
M
:
+91 99309 80509
O
:
+91 (22) 2687 9402
A
:
D Wing, 7th floor, 32 Corporate Avenue, Off Mahakali Caves Road, Andheri (E), Mumbai, 400093
W
:
www.unotechsoft.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.unotechsoft.com_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=6oi1htUxdeiGO6J_2mbMkxeZudlH1ZS-t4D0H5zA2VQ&s=4sIYPws_WQ4azv3GShryRfttnaxmcY-22IDBVQsP_C0&e=>
Disclaimer : The contents of this e-mail and attachment(s) thereto are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or Unotech Software Pvt. Ltd. or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of Unotech Software Pvt. Ltd. or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Re: Slow writes and Frequent timeouts
Posted by Akshay Suresh <ak...@unotechsoft.com>.
Hi
It turned out to be an issue in my yaml file - Each record had a size in
GB. I modified my yaml and I am getting
good results.
Thanks for the support.
Cheers.
On Tue, Apr 18, 2017 at 11:45 AM, Akshay Suresh <
akshay.suresh@unotechsoft.com> wrote:
> Hi
>
>
> - Hardware Specs
> RAM: 14GB
> Cores: 8
> Disk: 440GB SSD
>
> - I have deployed 7 instances ( Earlier, I mentioned 8 but I had to
> decommission 1 due to low disk space ) on AWS.
>
> - Every node is in the same datacenter.
> .
> - I have created a custom schema with 28 columns. Created a simple YAML
> file for the stress tool.
> The command is as follows:
> ./cassandra-stress user profile=foo.yml duration=180m ops\(insert=1\)
> cl=one -rate threads=850
>
> - I have used a heap space of 7GB. I am using the default CMS for GC.
>
> I am clearing all the data and logs. I will try to replicate the issue and
> post the exact metrics.
>
>
>
>
> Thanks in advance.
>
>
> On Mon, Apr 17, 2017 at 10:28 PM, Jon Haddad <jo...@gmail.com>
> wrote:
>
>> What are your hardware specs? Where are you running the cluster? Is
>> every node in the same physical datacenter? What command are you using to
>> run stress?
>>
>> On Apr 17, 2017, at 9:57 AM, Akshay Suresh <ak...@unotechsoft.com>
>> wrote:
>>
>> Hi
>>
>> I have not done much. Just created a schema with SimpleStrategy and a
>> replication factor of 3.
>> Then I created a yaml file
>> Now I am running the cassandra stress tool invoking the yaml file - with
>> 10,000 records ( no warmup ) with 10 concurrent threads. I am just running
>> writes ( no reads )
>>
>>
>> When I keep running this script again and again eventually my nodes start
>> going down and I start getting timeouts and out of memory errors.
>>
>>
>>
>> On Mon, Apr 17, 2017 at 9:23 PM, Hannu Kröger <hk...@gmail.com> wrote:
>>
>>> It would help to know what kind queries are slow.
>>>
>>> Hannu
>>>
>>> On 17 Apr 2017, at 18:42, Akshay Suresh <ak...@unotechsoft.com>
>>> wrote:
>>>
>>> Hi
>>>
>>> I have set up a cassandra cluster of 8 nodes.
>>>
>>> I am using Apache Cassandra 3.9
>>>
>>> While using cassandra-stress tool for load testing, I am getting really
>>> slow writes ( low upto few 10-20 writes per second ) along with frequent
>>> timeouts and out of heap space errors.
>>>
>>> Kindly let me know how do I resolve this issue.
>>>
>>> *Disclaimer :* The contents of this e-mail and attachment(s) thereto
>>> are confidential and intended for the named recipient(s) only. It shall not
>>> attach any liability on the originator or Unotech Software Pvt. Ltd. or its
>>> affiliates. Any views or opinions presented in this email are solely those
>>> of the author and may not necessarily reflect the opinions of Unotech
>>> Software Pvt. Ltd. or its affiliates. Any form of reproduction,
>>> dissemination, copying, disclosure, modification, distribution and / or
>>> publication of this message without the prior written consent of the author
>>> of this e-mail is strictly prohibited. If you have received this email in
>>> error please delete it and notify the sender immediately.
>>>
>>>
>>>
>>
>>
>> --
>> Regards,
>>
>>
>> *Akshay Suresh*
>> *Unotech Software Pvt. Ltd*
>>
>>
>> * <http://www.unotechsoft.com/>*
>> M : +91 99309 80509
>> O : +91 (22) 2687 9402
>> A : D Wing, 7th floor, 32 Corporate Avenue, Off Mahakali Caves Road,
>> Andheri (E), Mumbai, 400093
>> W : www.unotechsoft.com
>>
>> *Disclaimer :* The contents of this e-mail and attachment(s) thereto are
>> confidential and intended for the named recipient(s) only. It shall not
>> attach any liability on the originator or Unotech Software Pvt. Ltd. or its
>> affiliates. Any views or opinions presented in this email are solely those
>> of the author and may not necessarily reflect the opinions of Unotech
>> Software Pvt. Ltd. or its affiliates. Any form of reproduction,
>> dissemination, copying, disclosure, modification, distribution and / or
>> publication of this message without the prior written consent of the author
>> of this e-mail is strictly prohibited. If you have received this email in
>> error please delete it and notify the sender immediately.
>>
>>
>>
>
>
> --
> Regards,
>
>
> *Akshay Suresh*
> *Unotech Software Pvt. Ltd*
>
>
> * <http://www.unotechsoft.com/>*
> M : +91 99309 80509
> O : +91 (22) 2687 9402
> A : D Wing, 7th floor, 32 Corporate Avenue, Off Mahakali Caves Road,
> Andheri (E), Mumbai, 400093
> W : www.unotechsoft.com
>
--
Regards,
*Akshay Suresh*
*Unotech Software Pvt. Ltd*
* <http://www.unotechsoft.com/>*
M : +91 99309 80509
O : +91 (22) 2687 9402
A : D Wing, 7th floor, 32 Corporate Avenue, Off Mahakali Caves Road,
Andheri (E), Mumbai, 400093
W : www.unotechsoft.com
--
*Disclaimer :* The contents of this e-mail and attachment(s) thereto are
confidential and intended for the named recipient(s) only. It shall not
attach any liability on the originator or Unotech Software Pvt. Ltd. or its
affiliates. Any views or opinions presented in this email are solely those
of the author and may not necessarily reflect the opinions of Unotech
Software Pvt. Ltd. or its affiliates. Any form of reproduction,
dissemination, copying, disclosure, modification, distribution and / or
publication of this message without the prior written consent of the author
of this e-mail is strictly prohibited. If you have received this email in
error please delete it and notify the sender immediately.
Re: Slow writes and Frequent timeouts
Posted by Akshay Suresh <ak...@unotechsoft.com>.
Hi
- Hardware Specs
RAM: 14GB
Cores: 8
Disk: 440GB SSD
- I have deployed 7 instances ( Earlier, I mentioned 8 but I had to
decommission 1 due to low disk space ) on AWS.
- Every node is in the same datacenter.
.
- I have created a custom schema with 28 columns. Created a simple YAML
file for the stress tool.
The command is as follows:
./cassandra-stress user profile=foo.yml duration=180m ops\(insert=1\)
cl=one -rate threads=850
- I have used a heap space of 7GB. I am using the default CMS for GC.
I am clearing all the data and logs. I will try to replicate the issue and
post the exact metrics.
Thanks in advance.
On Mon, Apr 17, 2017 at 10:28 PM, Jon Haddad <jo...@gmail.com>
wrote:
> What are your hardware specs? Where are you running the cluster? Is
> every node in the same physical datacenter? What command are you using to
> run stress?
>
> On Apr 17, 2017, at 9:57 AM, Akshay Suresh <ak...@unotechsoft.com>
> wrote:
>
> Hi
>
> I have not done much. Just created a schema with SimpleStrategy and a
> replication factor of 3.
> Then I created a yaml file
> Now I am running the cassandra stress tool invoking the yaml file - with
> 10,000 records ( no warmup ) with 10 concurrent threads. I am just running
> writes ( no reads )
>
>
> When I keep running this script again and again eventually my nodes start
> going down and I start getting timeouts and out of memory errors.
>
>
>
> On Mon, Apr 17, 2017 at 9:23 PM, Hannu Kröger <hk...@gmail.com> wrote:
>
>> It would help to know what kind queries are slow.
>>
>> Hannu
>>
>> On 17 Apr 2017, at 18:42, Akshay Suresh <ak...@unotechsoft.com>
>> wrote:
>>
>> Hi
>>
>> I have set up a cassandra cluster of 8 nodes.
>>
>> I am using Apache Cassandra 3.9
>>
>> While using cassandra-stress tool for load testing, I am getting really
>> slow writes ( low upto few 10-20 writes per second ) along with frequent
>> timeouts and out of heap space errors.
>>
>> Kindly let me know how do I resolve this issue.
>>
>> *Disclaimer :* The contents of this e-mail and attachment(s) thereto are
>> confidential and intended for the named recipient(s) only. It shall not
>> attach any liability on the originator or Unotech Software Pvt. Ltd. or its
>> affiliates. Any views or opinions presented in this email are solely those
>> of the author and may not necessarily reflect the opinions of Unotech
>> Software Pvt. Ltd. or its affiliates. Any form of reproduction,
>> dissemination, copying, disclosure, modification, distribution and / or
>> publication of this message without the prior written consent of the author
>> of this e-mail is strictly prohibited. If you have received this email in
>> error please delete it and notify the sender immediately.
>>
>>
>>
>
>
> --
> Regards,
>
>
> *Akshay Suresh*
> *Unotech Software Pvt. Ltd*
>
>
> * <http://www.unotechsoft.com/>*
> M : +91 99309 80509
> O : +91 (22) 2687 9402
> A : D Wing, 7th floor, 32 Corporate Avenue, Off Mahakali Caves Road,
> Andheri (E), Mumbai, 400093
> W : www.unotechsoft.com
>
> *Disclaimer :* The contents of this e-mail and attachment(s) thereto are
> confidential and intended for the named recipient(s) only. It shall not
> attach any liability on the originator or Unotech Software Pvt. Ltd. or its
> affiliates. Any views or opinions presented in this email are solely those
> of the author and may not necessarily reflect the opinions of Unotech
> Software Pvt. Ltd. or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification, distribution and / or
> publication of this message without the prior written consent of the author
> of this e-mail is strictly prohibited. If you have received this email in
> error please delete it and notify the sender immediately.
>
>
>
--
Regards,
*Akshay Suresh*
*Unotech Software Pvt. Ltd*
* <http://www.unotechsoft.com/>*
M : +91 99309 80509
O : +91 (22) 2687 9402
A : D Wing, 7th floor, 32 Corporate Avenue, Off Mahakali Caves Road,
Andheri (E), Mumbai, 400093
W : www.unotechsoft.com
--
*Disclaimer :* The contents of this e-mail and attachment(s) thereto are
confidential and intended for the named recipient(s) only. It shall not
attach any liability on the originator or Unotech Software Pvt. Ltd. or its
affiliates. Any views or opinions presented in this email are solely those
of the author and may not necessarily reflect the opinions of Unotech
Software Pvt. Ltd. or its affiliates. Any form of reproduction,
dissemination, copying, disclosure, modification, distribution and / or
publication of this message without the prior written consent of the author
of this e-mail is strictly prohibited. If you have received this email in
error please delete it and notify the sender immediately.
Re: Slow writes and Frequent timeouts
Posted by Jon Haddad <jo...@gmail.com>.
What are your hardware specs? Where are you running the cluster? Is every node in the same physical datacenter? What command are you using to run stress?
> On Apr 17, 2017, at 9:57 AM, Akshay Suresh <ak...@unotechsoft.com> wrote:
>
> Hi
>
> I have not done much. Just created a schema with SimpleStrategy and a replication factor of 3.
> Then I created a yaml file
> Now I am running the cassandra stress tool invoking the yaml file - with 10,000 records ( no warmup ) with 10 concurrent threads. I am just running writes ( no reads )
>
>
> When I keep running this script again and again eventually my nodes start going down and I start getting timeouts and out of memory errors.
>
>
>
> On Mon, Apr 17, 2017 at 9:23 PM, Hannu Kröger <hkroger@gmail.com <ma...@gmail.com>> wrote:
> It would help to know what kind queries are slow.
>
> Hannu
>
>> On 17 Apr 2017, at 18:42, Akshay Suresh <akshay.suresh@unotechsoft.com <ma...@unotechsoft.com>> wrote:
>>
>> Hi
>>
>> I have set up a cassandra cluster of 8 nodes.
>>
>> I am using Apache Cassandra 3.9
>>
>> While using cassandra-stress tool for load testing, I am getting really slow writes ( low upto few 10-20 writes per second ) along with frequent timeouts and out of heap space errors.
>>
>> Kindly let me know how do I resolve this issue.
>>
>>
>> Disclaimer : The contents of this e-mail and attachment(s) thereto are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or Unotech Software Pvt. Ltd. or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of Unotech Software Pvt. Ltd. or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
>
>
>
>
> --
> Regards,
>
>
> Akshay Suresh
> Unotech Software Pvt. Ltd
> <http://www.unotechsoft.com/>
> M : +91 99309 80509
> O : +91 (22) 2687 9402
> A : D Wing, 7th floor, 32 Corporate Avenue, Off Mahakali Caves Road, Andheri (E), Mumbai, 400093
> W : www.unotechsoft.com <http://www.unotechsoft.com/>
> Disclaimer : The contents of this e-mail and attachment(s) thereto are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or Unotech Software Pvt. Ltd. or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of Unotech Software Pvt. Ltd. or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Re: Slow writes and Frequent timeouts
Posted by Akshay Suresh <ak...@unotechsoft.com>.
Hi
I have not done much. Just created a schema with SimpleStrategy and a
replication factor of 3.
Then I created a yaml file
Now I am running the cassandra stress tool invoking the yaml file - with
10,000 records ( no warmup ) with 10 concurrent threads. I am just running
writes ( no reads )
When I keep running this script again and again eventually my nodes start
going down and I start getting timeouts and out of memory errors.
On Mon, Apr 17, 2017 at 9:23 PM, Hannu Kröger <hk...@gmail.com> wrote:
> It would help to know what kind queries are slow.
>
> Hannu
>
> On 17 Apr 2017, at 18:42, Akshay Suresh <ak...@unotechsoft.com>
> wrote:
>
> Hi
>
> I have set up a cassandra cluster of 8 nodes.
>
> I am using Apache Cassandra 3.9
>
> While using cassandra-stress tool for load testing, I am getting really
> slow writes ( low upto few 10-20 writes per second ) along with frequent
> timeouts and out of heap space errors.
>
> Kindly let me know how do I resolve this issue.
>
> *Disclaimer :* The contents of this e-mail and attachment(s) thereto are
> confidential and intended for the named recipient(s) only. It shall not
> attach any liability on the originator or Unotech Software Pvt. Ltd. or its
> affiliates. Any views or opinions presented in this email are solely those
> of the author and may not necessarily reflect the opinions of Unotech
> Software Pvt. Ltd. or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification, distribution and / or
> publication of this message without the prior written consent of the author
> of this e-mail is strictly prohibited. If you have received this email in
> error please delete it and notify the sender immediately.
>
>
>
--
Regards,
*Akshay Suresh*
*Unotech Software Pvt. Ltd*
* <http://www.unotechsoft.com/>*
M : +91 99309 80509
O : +91 (22) 2687 9402
A : D Wing, 7th floor, 32 Corporate Avenue, Off Mahakali Caves Road,
Andheri (E), Mumbai, 400093
W : www.unotechsoft.com
--
*Disclaimer :* The contents of this e-mail and attachment(s) thereto are
confidential and intended for the named recipient(s) only. It shall not
attach any liability on the originator or Unotech Software Pvt. Ltd. or its
affiliates. Any views or opinions presented in this email are solely those
of the author and may not necessarily reflect the opinions of Unotech
Software Pvt. Ltd. or its affiliates. Any form of reproduction,
dissemination, copying, disclosure, modification, distribution and / or
publication of this message without the prior written consent of the author
of this e-mail is strictly prohibited. If you have received this email in
error please delete it and notify the sender immediately.
Re: Slow writes and Frequent timeouts
Posted by Hannu Kröger <hk...@gmail.com>.
It would help to know what kind queries are slow.
Hannu
> On 17 Apr 2017, at 18:42, Akshay Suresh <ak...@unotechsoft.com> wrote:
>
> Hi
>
> I have set up a cassandra cluster of 8 nodes.
>
> I am using Apache Cassandra 3.9
>
> While using cassandra-stress tool for load testing, I am getting really slow writes ( low upto few 10-20 writes per second ) along with frequent timeouts and out of heap space errors.
>
> Kindly let me know how do I resolve this issue.
>
>
> Disclaimer : The contents of this e-mail and attachment(s) thereto are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or Unotech Software Pvt. Ltd. or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of Unotech Software Pvt. Ltd. or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.