You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Aditya Shetty <ad...@snapdeal.com> on 2015/06/29 20:06:39 UTC
Cassandra write timeout
Hi
I have a 3 node cluster with a simple column family that stores product
information. However, even with low write loads, I get
WriteTimeoutExceptions when writing into it.
Im doing around 10 writes per second.
Strangely, when I check the database, all the products that had timed out
are actually present in the database. I dont think this is because of
hinted handoffs. I was monitoring the cluster the whole time and none of
the nodes went down.
Also, as per nodetool status the data is not equally distributed across all
3 nodes (even though they are identical in terms of config)
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host
ID Rack
UN 10.1.16.131 48.57 MB 256 ?
5aee18b0-7a75-478a-8648-2f2b1c6821fd rack1
UN 10.1.18.107 54.72 MB 256 ?
c307526c-e362-4b61-abc4-a2e951e38b86 rack1
UN 10.1.24.73 4.25 MB 256 ?
b10cb9ac-fbb1-4014-9534-26477d31fffd rack1
--
Aditya Shetty
*Lead Engineer*
*M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
2nd FLOOR, WEST WING,
SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
Download Our App [image: A]
<https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
[image: A]
<https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
[image: W]
<http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
Re: Cassandra write timeout
Posted by graham sanderson <gr...@vast.com>.
You should upgrade to 2.1.7 due to https://issues.apache.org/jira/browse/CASSANDRA-9549 <https://issues.apache.org/jira/browse/CASSANDRA-9549>
> On Jun 29, 2015, at 2:14 PM, Aditya Shetty <ad...@snapdeal.com> wrote:
>
> Hi Carlos
>
> I was mistaken earlier, I cross checked now, and found that the DevOps team had installed a later version, even though we asked for 2.1.3, the version installed on the server is 2.1.5.
>
> Do you think upgrading to 2.1.6 would be useful?
>
> On Tue, Jun 30, 2015 at 12:36 AM, Carlos Rolo <rolo@pythian.com <ma...@pythian.com>> wrote:
> Hello,
>
> Compaction issues are a know issue in the 2.1 series until 2.1.5.You can probably search this mailing list for further details and/or similar situations. And 2.1.3 is not considered a stable release. It is a good rule to always follow DSE releases as a guideline for which version to pick.
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: cjrolo | Linkedin: linkedin.com/in/carlosjuzarterolo <http://linkedin.com/in/carlosjuzarterolo>
> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
> www.pythian.com <http://www.pythian.com/>
> On Mon, Jun 29, 2015 at 8:53 PM, Aditya Shetty <aditya.shetty@snapdeal.com <ma...@snapdeal.com>> wrote:
> Hi Carlos
>
> Thanks for the tip. But is this a known issue in 2.1.3?
>
> On Tue, Jun 30, 2015 at 12:20 AM, Carlos Rolo <rolo@pythian.com <ma...@pythian.com>> wrote:
> Ok, as a first step I would recommend an upgrade to at least 2.1.6 (Or 2.17 since it is already the stable version). Given the size of your cluster should be a fairly easy operation (Upgrade and rolling restart) for more information follow the documentation: http://docs.datastax.com/en/upgrade/doc/upgrade/cassandra/upgradeCassandraDetails.html <http://docs.datastax.com/en/upgrade/doc/upgrade/cassandra/upgradeCassandraDetails.html>
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: cjrolo | Linkedin: linkedin.com/in/carlosjuzarterolo <http://linkedin.com/in/carlosjuzarterolo>
> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649 <tel:%2B1%20613%20565%208696%20x1649>
> www.pythian.com <http://www.pythian.com/>
> On Mon, Jun 29, 2015 at 8:45 PM, Aditya Shetty <aditya.shetty@snapdeal.com <ma...@snapdeal.com>> wrote:
> The cassandra version Im using is : 2.1.3
>
> On Tue, Jun 30, 2015 at 12:06 AM, Aditya Shetty <aditya.shetty@snapdeal.com <ma...@snapdeal.com>> wrote:
> Hi Carlos
>
> The node with 5MB does have compactions running while the others don't:
>
> pending tasks: 1
> compaction type keyspace table completed total unit progress
> Compaction reviews_platform object_stats 295 1300550330 bytes 0.00%
> Active compaction remaining time : 0h01m17s
>
>
> Also, the progress stays at 0%
>
>
> On other nodes:
> pending tasks: 0
>
> On Mon, Jun 29, 2015 at 11:54 PM, Carlos Rolo <rolo@pythian.com <ma...@pythian.com>> wrote:
> Hello,
>
> Can you please tell what is the version you are running? What is the Replication Factor and if the node with 5MB has compactions running (nodetool compactionstats)?
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: cjrolo | Linkedin: linkedin.com/in/carlosjuzarterolo <http://linkedin.com/in/carlosjuzarterolo>
> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649 <tel:%2B1%20613%20565%208696%20x1649>
> www.pythian.com <http://www.pythian.com/>
>
> On Mon, Jun 29, 2015 at 8:20 PM, Aditya Shetty <aditya.shetty@snapdeal.com <ma...@snapdeal.com>> wrote:
> Another interesting thing is that the node that has only 5MB if data (10.1.24.73) is the one where CPU utilization is quite high (about 100%) while the other nodes are at < 5%
>
> On Mon, Jun 29, 2015 at 11:36 PM, Aditya Shetty <aditya.shetty@snapdeal.com <ma...@snapdeal.com>> wrote:
> Hi
>
> I have a 3 node cluster with a simple column family that stores product information. However, even with low write loads, I get WriteTimeoutExceptions when writing into it.
>
> Im doing around 10 writes per second.
>
> Strangely, when I check the database, all the products that had timed out are actually present in the database. I dont think this is because of hinted handoffs. I was monitoring the cluster the whole time and none of the nodes went down.
>
> Also, as per nodetool status the data is not equally distributed across all 3 nodes (even though they are identical in terms of config)
>
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns Host ID Rack
> UN 10.1.16.131 48.57 MB 256 ? 5aee18b0-7a75-478a-8648-2f2b1c6821fd rack1
> UN 10.1.18.107 54.72 MB 256 ? c307526c-e362-4b61-abc4-a2e951e38b86 rack1
> UN 10.1.24.73 4.25 MB 256 ? b10cb9ac-fbb1-4014-9534-26477d31fffd rack1
>
>
> --
>
>
> Aditya Shetty
> Lead Engineer
>
>
> M: +91 7022423545 <tel:%2B91%207022423545>, T: 080 46603000 EXT: 4417
> 2nd FLOOR, WEST WING,
> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>
>
>
> Download Our App
> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>
>
>
> --
>
>
> Aditya Shetty
> Lead Engineer
>
>
> M: +91 7022423545 <tel:%2B91%207022423545>, T: 080 46603000 EXT: 4417
> 2nd FLOOR, WEST WING,
> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>
>
>
> Download Our App
> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>
>
> --
>
>
>
>
>
>
>
>
> --
>
>
> Aditya Shetty
> Lead Engineer
>
>
> M: +91 7022423545 <tel:%2B91%207022423545>, T: 080 46603000 EXT: 4417
> 2nd FLOOR, WEST WING,
> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>
>
>
> Download Our App
> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>
>
>
> --
>
>
> Aditya Shetty
> Lead Engineer
>
>
> M: +91 7022423545 <tel:%2B91%207022423545>, T: 080 46603000 EXT: 4417
> 2nd FLOOR, WEST WING,
> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>
>
>
> Download Our App
> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>
>
> --
>
>
>
>
>
>
>
>
> --
>
>
> Aditya Shetty
> Lead Engineer
>
>
> M: +91 7022423545 <tel:%2B91%207022423545>, T: 080 46603000 EXT: 4417
> 2nd FLOOR, WEST WING,
> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>
>
>
> Download Our App
> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>
>
> --
>
>
>
>
>
>
>
>
> --
>
>
> Aditya Shetty
> Lead Engineer
>
>
> M: +91 7022423545, T: 080 46603000 EXT: 4417
> 2nd FLOOR, WEST WING,
> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>
>
>
> Download Our App
> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
Re: Cassandra write timeout
Posted by Aditya Shetty <ad...@snapdeal.com>.
Hi Carlos
I was mistaken earlier, I cross checked now, and found that the DevOps
team had installed a later version, even though we asked for 2.1.3, the
version installed on the server is 2.1.5.
Do you think upgrading to 2.1.6 would be useful?
On Tue, Jun 30, 2015 at 12:36 AM, Carlos Rolo <ro...@pythian.com> wrote:
> Hello,
>
> Compaction issues are a know issue in the 2.1 series until 2.1.5.You can
> probably search this mailing list for further details and/or similar
> situations. And 2.1.3 is not considered a stable release. It is a good rule
> to always follow DSE releases as a guideline for which version to pick.
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
> <http://linkedin.com/in/carlosjuzarterolo>*
> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
> www.pythian.com
>
> On Mon, Jun 29, 2015 at 8:53 PM, Aditya Shetty <aditya.shetty@snapdeal.com
> > wrote:
>
>> Hi Carlos
>>
>> Thanks for the tip. But is this a known issue in 2.1.3?
>>
>> On Tue, Jun 30, 2015 at 12:20 AM, Carlos Rolo <ro...@pythian.com> wrote:
>>
>>> Ok, as a first step I would recommend an upgrade to at least 2.1.6 (Or
>>> 2.17 since it is already the stable version). Given the size of your
>>> cluster should be a fairly easy operation (Upgrade and rolling restart) for
>>> more information follow the documentation:
>>> http://docs.datastax.com/en/upgrade/doc/upgrade/cassandra/upgradeCassandraDetails.html
>>>
>>> Regards,
>>>
>>> Carlos Juzarte Rolo
>>> Cassandra Consultant
>>>
>>> Pythian - Love your data
>>>
>>> rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
>>> <http://linkedin.com/in/carlosjuzarterolo>*
>>> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
>>> www.pythian.com
>>>
>>> On Mon, Jun 29, 2015 at 8:45 PM, Aditya Shetty <
>>> aditya.shetty@snapdeal.com> wrote:
>>>
>>>> The cassandra version Im using is : 2.1.3
>>>>
>>>> On Tue, Jun 30, 2015 at 12:06 AM, Aditya Shetty <
>>>> aditya.shetty@snapdeal.com> wrote:
>>>>
>>>>> Hi Carlos
>>>>>
>>>>> The node with 5MB does have compactions running while the others don't:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *pending tasks: 1 compaction type keyspace
>>>>> table completed total unit progress Compaction
>>>>> reviews_platform object_stats 295 1300550330 bytes
>>>>> 0.00%Active compaction remaining time : 0h01m17s*
>>>>>
>>>>>
>>>>> Also, the progress stays at 0%
>>>>>
>>>>>
>>>>> On other nodes:
>>>>> *pending tasks: 0*
>>>>>
>>>>> On Mon, Jun 29, 2015 at 11:54 PM, Carlos Rolo <ro...@pythian.com>
>>>>> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Can you please tell what is the version you are running? What is the
>>>>>> Replication Factor and if the node with 5MB has compactions running
>>>>>> (nodetool compactionstats)?
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Carlos Juzarte Rolo
>>>>>> Cassandra Consultant
>>>>>>
>>>>>> Pythian - Love your data
>>>>>>
>>>>>> rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
>>>>>> <http://linkedin.com/in/carlosjuzarterolo>*
>>>>>> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
>>>>>> www.pythian.com
>>>>>>
>>>>>> On Mon, Jun 29, 2015 at 8:20 PM, Aditya Shetty <
>>>>>> aditya.shetty@snapdeal.com> wrote:
>>>>>>
>>>>>>> Another interesting thing is that the node that has only 5MB if data
>>>>>>> (10.1.24.73) is the one where CPU utilization is quite high (about 100%)
>>>>>>> while the other nodes are at < 5%
>>>>>>>
>>>>>>> On Mon, Jun 29, 2015 at 11:36 PM, Aditya Shetty <
>>>>>>> aditya.shetty@snapdeal.com> wrote:
>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> I have a 3 node cluster with a simple column family that stores
>>>>>>>> product information. However, even with low write loads, I get
>>>>>>>> WriteTimeoutExceptions when writing into it.
>>>>>>>>
>>>>>>>> Im doing around 10 writes per second.
>>>>>>>>
>>>>>>>> Strangely, when I check the database, all the products that had
>>>>>>>> timed out are actually present in the database. I dont think this is
>>>>>>>> because of hinted handoffs. I was monitoring the cluster the whole time and
>>>>>>>> none of the nodes went down.
>>>>>>>>
>>>>>>>> Also, as per nodetool status the data is not equally distributed
>>>>>>>> across all 3 nodes (even though they are identical in terms of config)
>>>>>>>>
>>>>>>>> Datacenter: datacenter1
>>>>>>>> =======================
>>>>>>>> Status=Up/Down
>>>>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>>>>> -- Address Load Tokens Owns Host
>>>>>>>> ID Rack
>>>>>>>> UN 10.1.16.131 48.57 MB 256 ?
>>>>>>>> 5aee18b0-7a75-478a-8648-2f2b1c6821fd rack1
>>>>>>>> UN 10.1.18.107 54.72 MB 256 ?
>>>>>>>> c307526c-e362-4b61-abc4-a2e951e38b86 rack1
>>>>>>>> UN 10.1.24.73 4.25 MB 256 ?
>>>>>>>> b10cb9ac-fbb1-4014-9534-26477d31fffd rack1
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Aditya Shetty
>>>>>>>> *Lead Engineer*
>>>>>>>>
>>>>>>>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>>>>>>>> 2nd FLOOR, WEST WING,
>>>>>>>>
>>>>>>>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>>>>>>>
>>>>>>>> Download Our App [image: A]
>>>>>>>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>>>>>>>> [image: A]
>>>>>>>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>>>>>>>> [image: W]
>>>>>>>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Aditya Shetty
>>>>>>> *Lead Engineer*
>>>>>>>
>>>>>>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>>>>>>> 2nd FLOOR, WEST WING,
>>>>>>>
>>>>>>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>>>>>>
>>>>>>> Download Our App [image: A]
>>>>>>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>>>>>>> [image: A]
>>>>>>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>>>>>>> [image: W]
>>>>>>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Aditya Shetty
>>>>> *Lead Engineer*
>>>>>
>>>>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>>>>> 2nd FLOOR, WEST WING,
>>>>>
>>>>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>>>>
>>>>> Download Our App [image: A]
>>>>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>>>>> [image: A]
>>>>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>>>>> [image: W]
>>>>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Aditya Shetty
>>>> *Lead Engineer*
>>>>
>>>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>>>> 2nd FLOOR, WEST WING,
>>>>
>>>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>>>
>>>> Download Our App [image: A]
>>>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>>>> [image: A]
>>>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>>>> [image: W]
>>>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>>>
>>>
>>>
>>> --
>>>
>>>
>>>
>>>
>>
>>
>> --
>>
>> Aditya Shetty
>> *Lead Engineer*
>>
>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>> 2nd FLOOR, WEST WING,
>>
>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>
>> Download Our App [image: A]
>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>> [image: A]
>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>> [image: W]
>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>
>
>
> --
>
>
>
>
--
Aditya Shetty
*Lead Engineer*
*M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
2nd FLOOR, WEST WING,
SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
Download Our App [image: A]
<https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
[image: A]
<https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
[image: W]
<http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
Re: Cassandra write timeout
Posted by Carlos Rolo <ro...@pythian.com>.
Hello,
Compaction issues are a know issue in the 2.1 series until 2.1.5.You can
probably search this mailing list for further details and/or similar
situations. And 2.1.3 is not considered a stable release. It is a good rule
to always follow DSE releases as a guideline for which version to pick.
Regards,
Carlos Juzarte Rolo
Cassandra Consultant
Pythian - Love your data
rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
<http://linkedin.com/in/carlosjuzarterolo>*
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.com
On Mon, Jun 29, 2015 at 8:53 PM, Aditya Shetty <ad...@snapdeal.com>
wrote:
> Hi Carlos
>
> Thanks for the tip. But is this a known issue in 2.1.3?
>
> On Tue, Jun 30, 2015 at 12:20 AM, Carlos Rolo <ro...@pythian.com> wrote:
>
>> Ok, as a first step I would recommend an upgrade to at least 2.1.6 (Or
>> 2.17 since it is already the stable version). Given the size of your
>> cluster should be a fairly easy operation (Upgrade and rolling restart) for
>> more information follow the documentation:
>> http://docs.datastax.com/en/upgrade/doc/upgrade/cassandra/upgradeCassandraDetails.html
>>
>> Regards,
>>
>> Carlos Juzarte Rolo
>> Cassandra Consultant
>>
>> Pythian - Love your data
>>
>> rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
>> <http://linkedin.com/in/carlosjuzarterolo>*
>> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
>> www.pythian.com
>>
>> On Mon, Jun 29, 2015 at 8:45 PM, Aditya Shetty <
>> aditya.shetty@snapdeal.com> wrote:
>>
>>> The cassandra version Im using is : 2.1.3
>>>
>>> On Tue, Jun 30, 2015 at 12:06 AM, Aditya Shetty <
>>> aditya.shetty@snapdeal.com> wrote:
>>>
>>>> Hi Carlos
>>>>
>>>> The node with 5MB does have compactions running while the others don't:
>>>>
>>>>
>>>>
>>>>
>>>> *pending tasks: 1 compaction type keyspace table
>>>> completed total unit progress Compaction
>>>> reviews_platform object_stats 295 1300550330 bytes
>>>> 0.00%Active compaction remaining time : 0h01m17s*
>>>>
>>>>
>>>> Also, the progress stays at 0%
>>>>
>>>>
>>>> On other nodes:
>>>> *pending tasks: 0*
>>>>
>>>> On Mon, Jun 29, 2015 at 11:54 PM, Carlos Rolo <ro...@pythian.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> Can you please tell what is the version you are running? What is the
>>>>> Replication Factor and if the node with 5MB has compactions running
>>>>> (nodetool compactionstats)?
>>>>>
>>>>> Regards,
>>>>>
>>>>> Carlos Juzarte Rolo
>>>>> Cassandra Consultant
>>>>>
>>>>> Pythian - Love your data
>>>>>
>>>>> rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
>>>>> <http://linkedin.com/in/carlosjuzarterolo>*
>>>>> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
>>>>> www.pythian.com
>>>>>
>>>>> On Mon, Jun 29, 2015 at 8:20 PM, Aditya Shetty <
>>>>> aditya.shetty@snapdeal.com> wrote:
>>>>>
>>>>>> Another interesting thing is that the node that has only 5MB if data
>>>>>> (10.1.24.73) is the one where CPU utilization is quite high (about 100%)
>>>>>> while the other nodes are at < 5%
>>>>>>
>>>>>> On Mon, Jun 29, 2015 at 11:36 PM, Aditya Shetty <
>>>>>> aditya.shetty@snapdeal.com> wrote:
>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> I have a 3 node cluster with a simple column family that stores
>>>>>>> product information. However, even with low write loads, I get
>>>>>>> WriteTimeoutExceptions when writing into it.
>>>>>>>
>>>>>>> Im doing around 10 writes per second.
>>>>>>>
>>>>>>> Strangely, when I check the database, all the products that had
>>>>>>> timed out are actually present in the database. I dont think this is
>>>>>>> because of hinted handoffs. I was monitoring the cluster the whole time and
>>>>>>> none of the nodes went down.
>>>>>>>
>>>>>>> Also, as per nodetool status the data is not equally distributed
>>>>>>> across all 3 nodes (even though they are identical in terms of config)
>>>>>>>
>>>>>>> Datacenter: datacenter1
>>>>>>> =======================
>>>>>>> Status=Up/Down
>>>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>>>> -- Address Load Tokens Owns Host
>>>>>>> ID Rack
>>>>>>> UN 10.1.16.131 48.57 MB 256 ?
>>>>>>> 5aee18b0-7a75-478a-8648-2f2b1c6821fd rack1
>>>>>>> UN 10.1.18.107 54.72 MB 256 ?
>>>>>>> c307526c-e362-4b61-abc4-a2e951e38b86 rack1
>>>>>>> UN 10.1.24.73 4.25 MB 256 ?
>>>>>>> b10cb9ac-fbb1-4014-9534-26477d31fffd rack1
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Aditya Shetty
>>>>>>> *Lead Engineer*
>>>>>>>
>>>>>>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>>>>>>> 2nd FLOOR, WEST WING,
>>>>>>>
>>>>>>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>>>>>>
>>>>>>> Download Our App [image: A]
>>>>>>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>>>>>>> [image: A]
>>>>>>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>>>>>>> [image: W]
>>>>>>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Aditya Shetty
>>>>>> *Lead Engineer*
>>>>>>
>>>>>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>>>>>> 2nd FLOOR, WEST WING,
>>>>>>
>>>>>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>>>>>
>>>>>> Download Our App [image: A]
>>>>>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>>>>>> [image: A]
>>>>>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>>>>>> [image: W]
>>>>>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Aditya Shetty
>>>> *Lead Engineer*
>>>>
>>>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>>>> 2nd FLOOR, WEST WING,
>>>>
>>>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>>>
>>>> Download Our App [image: A]
>>>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>>>> [image: A]
>>>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>>>> [image: W]
>>>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Aditya Shetty
>>> *Lead Engineer*
>>>
>>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>>> 2nd FLOOR, WEST WING,
>>>
>>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>>
>>> Download Our App [image: A]
>>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>>> [image: A]
>>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>>> [image: W]
>>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>>
>>
>>
>> --
>>
>>
>>
>>
>
>
> --
>
> Aditya Shetty
> *Lead Engineer*
>
> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
> 2nd FLOOR, WEST WING,
>
> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>
> Download Our App [image: A]
> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
> [image: A]
> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
> [image: W]
> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>
--
--
Re: Cassandra write timeout
Posted by Aditya Shetty <ad...@snapdeal.com>.
Hi Carlos
Thanks for the tip. But is this a known issue in 2.1.3?
On Tue, Jun 30, 2015 at 12:20 AM, Carlos Rolo <ro...@pythian.com> wrote:
> Ok, as a first step I would recommend an upgrade to at least 2.1.6 (Or
> 2.17 since it is already the stable version). Given the size of your
> cluster should be a fairly easy operation (Upgrade and rolling restart) for
> more information follow the documentation:
> http://docs.datastax.com/en/upgrade/doc/upgrade/cassandra/upgradeCassandraDetails.html
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
> <http://linkedin.com/in/carlosjuzarterolo>*
> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
> www.pythian.com
>
> On Mon, Jun 29, 2015 at 8:45 PM, Aditya Shetty <aditya.shetty@snapdeal.com
> > wrote:
>
>> The cassandra version Im using is : 2.1.3
>>
>> On Tue, Jun 30, 2015 at 12:06 AM, Aditya Shetty <
>> aditya.shetty@snapdeal.com> wrote:
>>
>>> Hi Carlos
>>>
>>> The node with 5MB does have compactions running while the others don't:
>>>
>>>
>>>
>>>
>>> *pending tasks: 1 compaction type keyspace table
>>> completed total unit progress Compaction
>>> reviews_platform object_stats 295 1300550330 bytes
>>> 0.00%Active compaction remaining time : 0h01m17s*
>>>
>>>
>>> Also, the progress stays at 0%
>>>
>>>
>>> On other nodes:
>>> *pending tasks: 0*
>>>
>>> On Mon, Jun 29, 2015 at 11:54 PM, Carlos Rolo <ro...@pythian.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> Can you please tell what is the version you are running? What is the
>>>> Replication Factor and if the node with 5MB has compactions running
>>>> (nodetool compactionstats)?
>>>>
>>>> Regards,
>>>>
>>>> Carlos Juzarte Rolo
>>>> Cassandra Consultant
>>>>
>>>> Pythian - Love your data
>>>>
>>>> rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
>>>> <http://linkedin.com/in/carlosjuzarterolo>*
>>>> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
>>>> www.pythian.com
>>>>
>>>> On Mon, Jun 29, 2015 at 8:20 PM, Aditya Shetty <
>>>> aditya.shetty@snapdeal.com> wrote:
>>>>
>>>>> Another interesting thing is that the node that has only 5MB if data
>>>>> (10.1.24.73) is the one where CPU utilization is quite high (about 100%)
>>>>> while the other nodes are at < 5%
>>>>>
>>>>> On Mon, Jun 29, 2015 at 11:36 PM, Aditya Shetty <
>>>>> aditya.shetty@snapdeal.com> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> I have a 3 node cluster with a simple column family that stores
>>>>>> product information. However, even with low write loads, I get
>>>>>> WriteTimeoutExceptions when writing into it.
>>>>>>
>>>>>> Im doing around 10 writes per second.
>>>>>>
>>>>>> Strangely, when I check the database, all the products that had timed
>>>>>> out are actually present in the database. I dont think this is because of
>>>>>> hinted handoffs. I was monitoring the cluster the whole time and none of
>>>>>> the nodes went down.
>>>>>>
>>>>>> Also, as per nodetool status the data is not equally distributed
>>>>>> across all 3 nodes (even though they are identical in terms of config)
>>>>>>
>>>>>> Datacenter: datacenter1
>>>>>> =======================
>>>>>> Status=Up/Down
>>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>>> -- Address Load Tokens Owns Host
>>>>>> ID Rack
>>>>>> UN 10.1.16.131 48.57 MB 256 ?
>>>>>> 5aee18b0-7a75-478a-8648-2f2b1c6821fd rack1
>>>>>> UN 10.1.18.107 54.72 MB 256 ?
>>>>>> c307526c-e362-4b61-abc4-a2e951e38b86 rack1
>>>>>> UN 10.1.24.73 4.25 MB 256 ?
>>>>>> b10cb9ac-fbb1-4014-9534-26477d31fffd rack1
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Aditya Shetty
>>>>>> *Lead Engineer*
>>>>>>
>>>>>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>>>>>> 2nd FLOOR, WEST WING,
>>>>>>
>>>>>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>>>>>
>>>>>> Download Our App [image: A]
>>>>>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>>>>>> [image: A]
>>>>>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>>>>>> [image: W]
>>>>>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Aditya Shetty
>>>>> *Lead Engineer*
>>>>>
>>>>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>>>>> 2nd FLOOR, WEST WING,
>>>>>
>>>>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>>>>
>>>>> Download Our App [image: A]
>>>>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>>>>> [image: A]
>>>>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>>>>> [image: W]
>>>>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Aditya Shetty
>>> *Lead Engineer*
>>>
>>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>>> 2nd FLOOR, WEST WING,
>>>
>>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>>
>>> Download Our App [image: A]
>>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>>> [image: A]
>>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>>> [image: W]
>>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>>
>>
>>
>>
>> --
>>
>> Aditya Shetty
>> *Lead Engineer*
>>
>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>> 2nd FLOOR, WEST WING,
>>
>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>
>> Download Our App [image: A]
>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>> [image: A]
>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>> [image: W]
>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>
>
>
> --
>
>
>
>
--
Aditya Shetty
*Lead Engineer*
*M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
2nd FLOOR, WEST WING,
SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
Download Our App [image: A]
<https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
[image: A]
<https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
[image: W]
<http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
Re: Cassandra write timeout
Posted by Carlos Rolo <ro...@pythian.com>.
Ok, as a first step I would recommend an upgrade to at least 2.1.6 (Or 2.17
since it is already the stable version). Given the size of your cluster
should be a fairly easy operation (Upgrade and rolling restart) for more
information follow the documentation:
http://docs.datastax.com/en/upgrade/doc/upgrade/cassandra/upgradeCassandraDetails.html
Regards,
Carlos Juzarte Rolo
Cassandra Consultant
Pythian - Love your data
rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
<http://linkedin.com/in/carlosjuzarterolo>*
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.com
On Mon, Jun 29, 2015 at 8:45 PM, Aditya Shetty <ad...@snapdeal.com>
wrote:
> The cassandra version Im using is : 2.1.3
>
> On Tue, Jun 30, 2015 at 12:06 AM, Aditya Shetty <
> aditya.shetty@snapdeal.com> wrote:
>
>> Hi Carlos
>>
>> The node with 5MB does have compactions running while the others don't:
>>
>>
>>
>>
>> *pending tasks: 1 compaction type keyspace table
>> completed total unit progress Compaction
>> reviews_platform object_stats 295 1300550330 bytes
>> 0.00%Active compaction remaining time : 0h01m17s*
>>
>>
>> Also, the progress stays at 0%
>>
>>
>> On other nodes:
>> *pending tasks: 0*
>>
>> On Mon, Jun 29, 2015 at 11:54 PM, Carlos Rolo <ro...@pythian.com> wrote:
>>
>>> Hello,
>>>
>>> Can you please tell what is the version you are running? What is the
>>> Replication Factor and if the node with 5MB has compactions running
>>> (nodetool compactionstats)?
>>>
>>> Regards,
>>>
>>> Carlos Juzarte Rolo
>>> Cassandra Consultant
>>>
>>> Pythian - Love your data
>>>
>>> rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
>>> <http://linkedin.com/in/carlosjuzarterolo>*
>>> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
>>> www.pythian.com
>>>
>>> On Mon, Jun 29, 2015 at 8:20 PM, Aditya Shetty <
>>> aditya.shetty@snapdeal.com> wrote:
>>>
>>>> Another interesting thing is that the node that has only 5MB if data
>>>> (10.1.24.73) is the one where CPU utilization is quite high (about 100%)
>>>> while the other nodes are at < 5%
>>>>
>>>> On Mon, Jun 29, 2015 at 11:36 PM, Aditya Shetty <
>>>> aditya.shetty@snapdeal.com> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> I have a 3 node cluster with a simple column family that stores
>>>>> product information. However, even with low write loads, I get
>>>>> WriteTimeoutExceptions when writing into it.
>>>>>
>>>>> Im doing around 10 writes per second.
>>>>>
>>>>> Strangely, when I check the database, all the products that had timed
>>>>> out are actually present in the database. I dont think this is because of
>>>>> hinted handoffs. I was monitoring the cluster the whole time and none of
>>>>> the nodes went down.
>>>>>
>>>>> Also, as per nodetool status the data is not equally distributed
>>>>> across all 3 nodes (even though they are identical in terms of config)
>>>>>
>>>>> Datacenter: datacenter1
>>>>> =======================
>>>>> Status=Up/Down
>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>> -- Address Load Tokens Owns Host
>>>>> ID Rack
>>>>> UN 10.1.16.131 48.57 MB 256 ?
>>>>> 5aee18b0-7a75-478a-8648-2f2b1c6821fd rack1
>>>>> UN 10.1.18.107 54.72 MB 256 ?
>>>>> c307526c-e362-4b61-abc4-a2e951e38b86 rack1
>>>>> UN 10.1.24.73 4.25 MB 256 ?
>>>>> b10cb9ac-fbb1-4014-9534-26477d31fffd rack1
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Aditya Shetty
>>>>> *Lead Engineer*
>>>>>
>>>>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>>>>> 2nd FLOOR, WEST WING,
>>>>>
>>>>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>>>>
>>>>> Download Our App [image: A]
>>>>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>>>>> [image: A]
>>>>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>>>>> [image: W]
>>>>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Aditya Shetty
>>>> *Lead Engineer*
>>>>
>>>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>>>> 2nd FLOOR, WEST WING,
>>>>
>>>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>>>
>>>> Download Our App [image: A]
>>>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>>>> [image: A]
>>>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>>>> [image: W]
>>>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>>>
>>>
>>>
>>> --
>>>
>>>
>>>
>>>
>>
>>
>> --
>>
>> Aditya Shetty
>> *Lead Engineer*
>>
>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>> 2nd FLOOR, WEST WING,
>>
>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>
>> Download Our App [image: A]
>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>> [image: A]
>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>> [image: W]
>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>
>
>
>
> --
>
> Aditya Shetty
> *Lead Engineer*
>
> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
> 2nd FLOOR, WEST WING,
>
> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>
> Download Our App [image: A]
> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
> [image: A]
> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
> [image: W]
> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>
--
--
Re: Cassandra write timeout
Posted by Aditya Shetty <ad...@snapdeal.com>.
The cassandra version Im using is : 2.1.3
On Tue, Jun 30, 2015 at 12:06 AM, Aditya Shetty <ad...@snapdeal.com>
wrote:
> Hi Carlos
>
> The node with 5MB does have compactions running while the others don't:
>
>
>
>
> *pending tasks: 1 compaction type keyspace table
> completed total unit progress Compaction
> reviews_platform object_stats 295 1300550330 bytes
> 0.00%Active compaction remaining time : 0h01m17s*
>
>
> Also, the progress stays at 0%
>
>
> On other nodes:
> *pending tasks: 0*
>
> On Mon, Jun 29, 2015 at 11:54 PM, Carlos Rolo <ro...@pythian.com> wrote:
>
>> Hello,
>>
>> Can you please tell what is the version you are running? What is the
>> Replication Factor and if the node with 5MB has compactions running
>> (nodetool compactionstats)?
>>
>> Regards,
>>
>> Carlos Juzarte Rolo
>> Cassandra Consultant
>>
>> Pythian - Love your data
>>
>> rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
>> <http://linkedin.com/in/carlosjuzarterolo>*
>> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
>> www.pythian.com
>>
>> On Mon, Jun 29, 2015 at 8:20 PM, Aditya Shetty <
>> aditya.shetty@snapdeal.com> wrote:
>>
>>> Another interesting thing is that the node that has only 5MB if data
>>> (10.1.24.73) is the one where CPU utilization is quite high (about 100%)
>>> while the other nodes are at < 5%
>>>
>>> On Mon, Jun 29, 2015 at 11:36 PM, Aditya Shetty <
>>> aditya.shetty@snapdeal.com> wrote:
>>>
>>>> Hi
>>>>
>>>> I have a 3 node cluster with a simple column family that stores product
>>>> information. However, even with low write loads, I get
>>>> WriteTimeoutExceptions when writing into it.
>>>>
>>>> Im doing around 10 writes per second.
>>>>
>>>> Strangely, when I check the database, all the products that had timed
>>>> out are actually present in the database. I dont think this is because of
>>>> hinted handoffs. I was monitoring the cluster the whole time and none of
>>>> the nodes went down.
>>>>
>>>> Also, as per nodetool status the data is not equally distributed across
>>>> all 3 nodes (even though they are identical in terms of config)
>>>>
>>>> Datacenter: datacenter1
>>>> =======================
>>>> Status=Up/Down
>>>> |/ State=Normal/Leaving/Joining/Moving
>>>> -- Address Load Tokens Owns Host
>>>> ID Rack
>>>> UN 10.1.16.131 48.57 MB 256 ?
>>>> 5aee18b0-7a75-478a-8648-2f2b1c6821fd rack1
>>>> UN 10.1.18.107 54.72 MB 256 ?
>>>> c307526c-e362-4b61-abc4-a2e951e38b86 rack1
>>>> UN 10.1.24.73 4.25 MB 256 ?
>>>> b10cb9ac-fbb1-4014-9534-26477d31fffd rack1
>>>>
>>>>
>>>> --
>>>>
>>>> Aditya Shetty
>>>> *Lead Engineer*
>>>>
>>>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>>>> 2nd FLOOR, WEST WING,
>>>>
>>>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>>>
>>>> Download Our App [image: A]
>>>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>>>> [image: A]
>>>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>>>> [image: W]
>>>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Aditya Shetty
>>> *Lead Engineer*
>>>
>>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>>> 2nd FLOOR, WEST WING,
>>>
>>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>>
>>> Download Our App [image: A]
>>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>>> [image: A]
>>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>>> [image: W]
>>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>>
>>
>>
>> --
>>
>>
>>
>>
>
>
> --
>
> Aditya Shetty
> *Lead Engineer*
>
> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
> 2nd FLOOR, WEST WING,
>
> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>
> Download Our App [image: A]
> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
> [image: A]
> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
> [image: W]
> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>
--
Aditya Shetty
*Lead Engineer*
*M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
2nd FLOOR, WEST WING,
SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
Download Our App [image: A]
<https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
[image: A]
<https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
[image: W]
<http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
Re: Cassandra write timeout
Posted by Aditya Shetty <ad...@snapdeal.com>.
Hi Carlos
The node with 5MB does have compactions running while the others don't:
*pending tasks: 1 compaction type keyspace table
completed total unit progress Compaction
reviews_platform object_stats 295 1300550330 bytes
0.00%Active compaction remaining time : 0h01m17s*
Also, the progress stays at 0%
On other nodes:
*pending tasks: 0*
On Mon, Jun 29, 2015 at 11:54 PM, Carlos Rolo <ro...@pythian.com> wrote:
> Hello,
>
> Can you please tell what is the version you are running? What is the
> Replication Factor and if the node with 5MB has compactions running
> (nodetool compactionstats)?
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
> <http://linkedin.com/in/carlosjuzarterolo>*
> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
> www.pythian.com
>
> On Mon, Jun 29, 2015 at 8:20 PM, Aditya Shetty <aditya.shetty@snapdeal.com
> > wrote:
>
>> Another interesting thing is that the node that has only 5MB if data
>> (10.1.24.73) is the one where CPU utilization is quite high (about 100%)
>> while the other nodes are at < 5%
>>
>> On Mon, Jun 29, 2015 at 11:36 PM, Aditya Shetty <
>> aditya.shetty@snapdeal.com> wrote:
>>
>>> Hi
>>>
>>> I have a 3 node cluster with a simple column family that stores product
>>> information. However, even with low write loads, I get
>>> WriteTimeoutExceptions when writing into it.
>>>
>>> Im doing around 10 writes per second.
>>>
>>> Strangely, when I check the database, all the products that had timed
>>> out are actually present in the database. I dont think this is because of
>>> hinted handoffs. I was monitoring the cluster the whole time and none of
>>> the nodes went down.
>>>
>>> Also, as per nodetool status the data is not equally distributed across
>>> all 3 nodes (even though they are identical in terms of config)
>>>
>>> Datacenter: datacenter1
>>> =======================
>>> Status=Up/Down
>>> |/ State=Normal/Leaving/Joining/Moving
>>> -- Address Load Tokens Owns Host
>>> ID Rack
>>> UN 10.1.16.131 48.57 MB 256 ?
>>> 5aee18b0-7a75-478a-8648-2f2b1c6821fd rack1
>>> UN 10.1.18.107 54.72 MB 256 ?
>>> c307526c-e362-4b61-abc4-a2e951e38b86 rack1
>>> UN 10.1.24.73 4.25 MB 256 ?
>>> b10cb9ac-fbb1-4014-9534-26477d31fffd rack1
>>>
>>>
>>> --
>>>
>>> Aditya Shetty
>>> *Lead Engineer*
>>>
>>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>>> 2nd FLOOR, WEST WING,
>>>
>>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>>
>>> Download Our App [image: A]
>>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>>> [image: A]
>>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>>> [image: W]
>>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>>
>>
>>
>>
>> --
>>
>> Aditya Shetty
>> *Lead Engineer*
>>
>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>> 2nd FLOOR, WEST WING,
>>
>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>
>> Download Our App [image: A]
>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>> [image: A]
>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>> [image: W]
>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>
>
>
> --
>
>
>
>
--
Aditya Shetty
*Lead Engineer*
*M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
2nd FLOOR, WEST WING,
SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
Download Our App [image: A]
<https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
[image: A]
<https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
[image: W]
<http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
Re: Cassandra write timeout
Posted by Carlos Rolo <ro...@pythian.com>.
Hello,
Can you please tell what is the version you are running? What is the
Replication Factor and if the node with 5MB has compactions running
(nodetool compactionstats)?
Regards,
Carlos Juzarte Rolo
Cassandra Consultant
Pythian - Love your data
rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
<http://linkedin.com/in/carlosjuzarterolo>*
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.com
On Mon, Jun 29, 2015 at 8:20 PM, Aditya Shetty <ad...@snapdeal.com>
wrote:
> Another interesting thing is that the node that has only 5MB if data
> (10.1.24.73) is the one where CPU utilization is quite high (about 100%)
> while the other nodes are at < 5%
>
> On Mon, Jun 29, 2015 at 11:36 PM, Aditya Shetty <
> aditya.shetty@snapdeal.com> wrote:
>
>> Hi
>>
>> I have a 3 node cluster with a simple column family that stores product
>> information. However, even with low write loads, I get
>> WriteTimeoutExceptions when writing into it.
>>
>> Im doing around 10 writes per second.
>>
>> Strangely, when I check the database, all the products that had timed out
>> are actually present in the database. I dont think this is because of
>> hinted handoffs. I was monitoring the cluster the whole time and none of
>> the nodes went down.
>>
>> Also, as per nodetool status the data is not equally distributed across
>> all 3 nodes (even though they are identical in terms of config)
>>
>> Datacenter: datacenter1
>> =======================
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns Host
>> ID Rack
>> UN 10.1.16.131 48.57 MB 256 ?
>> 5aee18b0-7a75-478a-8648-2f2b1c6821fd rack1
>> UN 10.1.18.107 54.72 MB 256 ?
>> c307526c-e362-4b61-abc4-a2e951e38b86 rack1
>> UN 10.1.24.73 4.25 MB 256 ?
>> b10cb9ac-fbb1-4014-9534-26477d31fffd rack1
>>
>>
>> --
>>
>> Aditya Shetty
>> *Lead Engineer*
>>
>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>> 2nd FLOOR, WEST WING,
>>
>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>
>> Download Our App [image: A]
>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>> [image: A]
>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>> [image: W]
>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>
>
>
>
> --
>
> Aditya Shetty
> *Lead Engineer*
>
> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
> 2nd FLOOR, WEST WING,
>
> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>
> Download Our App [image: A]
> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
> [image: A]
> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
> [image: W]
> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>
--
--
Re: Cassandra write timeout
Posted by Aditya Shetty <ad...@snapdeal.com>.
Another interesting thing is that the node that has only 5MB if data
(10.1.24.73) is the one where CPU utilization is quite high (about 100%)
while the other nodes are at < 5%
On Mon, Jun 29, 2015 at 11:36 PM, Aditya Shetty <ad...@snapdeal.com>
wrote:
> Hi
>
> I have a 3 node cluster with a simple column family that stores product
> information. However, even with low write loads, I get
> WriteTimeoutExceptions when writing into it.
>
> Im doing around 10 writes per second.
>
> Strangely, when I check the database, all the products that had timed out
> are actually present in the database. I dont think this is because of
> hinted handoffs. I was monitoring the cluster the whole time and none of
> the nodes went down.
>
> Also, as per nodetool status the data is not equally distributed across
> all 3 nodes (even though they are identical in terms of config)
>
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns Host
> ID Rack
> UN 10.1.16.131 48.57 MB 256 ?
> 5aee18b0-7a75-478a-8648-2f2b1c6821fd rack1
> UN 10.1.18.107 54.72 MB 256 ?
> c307526c-e362-4b61-abc4-a2e951e38b86 rack1
> UN 10.1.24.73 4.25 MB 256 ?
> b10cb9ac-fbb1-4014-9534-26477d31fffd rack1
>
>
> --
>
> Aditya Shetty
> *Lead Engineer*
>
> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
> 2nd FLOOR, WEST WING,
>
> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>
> Download Our App [image: A]
> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
> [image: A]
> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
> [image: W]
> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>
--
Aditya Shetty
*Lead Engineer*
*M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
2nd FLOOR, WEST WING,
SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
Download Our App [image: A]
<https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
[image: A]
<https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
[image: W]
<http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>