You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@cassandra.apache.org by Bhuvan Rawal <bh...@gmail.com> on 2016/06/22 13:03:58 UTC

Slow nodetool response time

Hi,

We have been facing slowness in getting response from nodetool for any of
its subcommand. On the same version on AWS it responds really fast but on
local 1 node machine or local DC cluster it performs very slow.

On Local DC :
*$ time nodetool version*
ReleaseVersion: 3.0.3

real 0m*17.582s*
user 0m2.334s
sys 0m0.470s

On AWS:
*$ time nodetool version*
ReleaseVersion: 3.0.3

real 0m*1.084s*
user 0m1.772s
sys 0m0.363s

Any way by which its speed can be increased?

Thanks & Regards,
Bhuvan

Re: Slow nodetool response time

Posted by Bhuvan Rawal <bh...@gmail.com>.

Thanks for your reply Sebastian. We have a 3 node dev cluster setup on
local servers. I tried nodetool commands on all 3 of them and response time
was same (excess of 17 Seconds) and most in blocked state as you found out.
I tried fiddling out with /etc/hosts file and on adding the line:

<private-ip> <hostname>

This change Made the wait nearly minmal and has possibly resolved the
problem. I can share the response on the same machine for which I had
earlier shared (it was greater than 17 secs earlier)
$ time nodetool version
ReleaseVersion: 3.0.3

real 0m*1.197s*
user 0m1.963s
sys 0m0.284s

This is a bit puzzling as I checked and ping is working in sub millisecond
from host to self and other two servers, wondering where that time was
being spent earlier.

Thanks & Regards,
Bhuvan

On Wed, Jun 22, 2016 at 7:10 PM, Sebastian Estevez <
sebastian.estevez@datastax.com> wrote:

> Sounds like your process is spending a lot of time in blocked state (real
> - user - sys). Check your os subsystems, maybe your machine is bogged down
> by other work.
>
> FWIW, my time in an idle system is about 2 seconds but can go up to ~13
> seconds on a busy system with 70% cpu utilized. No difference between 1 and
> 3 node setups.
>
> All the best,
>
>
> [image: datastax_logo.png] <http://www.datastax.com/>
>
> Sebastián Estévez
>
> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>
> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
> <https://twitter.com/datastax> [image: g+.png]
> <https://plus.google.com/+Datastax/about>
> <http://feeds.feedburner.com/datastax>
> <http://goog_410786983>
>
>
> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>
> DataStax is the fastest, most scalable distributed database technology,
> delivering Apache Cassandra to the world’s most innovative enterprises.
> Datastax is built to be agile, always-on, and predictably scalable to any
> size. With more than 500 customers in 45 countries, DataStax is the
> database technology and transactional backbone of choice for the worlds
> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>
> On Wed, Jun 22, 2016 at 9:03 AM, Bhuvan Rawal <bh...@gmail.com> wrote:
>
>> Hi,
>>
>> We have been facing slowness in getting response from nodetool for any of
>> its subcommand. On the same version on AWS it responds really fast but on
>> local 1 node machine or local DC cluster it performs very slow.
>>
>> On Local DC :
>> *$ time nodetool version*
>> ReleaseVersion: 3.0.3
>>
>> real 0m*17.582s*
>> user 0m2.334s
>> sys 0m0.470s
>>
>> On AWS:
>> *$ time nodetool version*
>> ReleaseVersion: 3.0.3
>>
>> real 0m*1.084s*
>> user 0m1.772s
>> sys 0m0.363s
>>
>> Any way by which its speed can be increased?
>>
>> Thanks & Regards,
>> Bhuvan
>>
>
>

Re: Slow nodetool response time

Posted by Sebastian Estevez <se...@datastax.com>.

Sounds like your process is spending a lot of time in blocked state (real -
user - sys). Check your os subsystems, maybe your machine is bogged down by
other work.

FWIW, my time in an idle system is about 2 seconds but can go up to ~13
seconds on a busy system with 70% cpu utilized. No difference between 1 and
3 node setups.

All the best,

[image: datastax_logo.png] <http://www.datastax.com/>

Sebastián Estévez

Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com

[image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
<https://twitter.com/datastax> [image: g+.png]
<https://plus.google.com/+Datastax/about>
<http://feeds.feedburner.com/datastax>
<http://goog_410786983>

<http://www.datastax.com/gartner-magic-quadrant-odbms>

DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.

On Wed, Jun 22, 2016 at 9:03 AM, Bhuvan Rawal <bh...@gmail.com> wrote:

> Hi,
>
> We have been facing slowness in getting response from nodetool for any of
> its subcommand. On the same version on AWS it responds really fast but on
> local 1 node machine or local DC cluster it performs very slow.
>
> On Local DC :
> *$ time nodetool version*
> ReleaseVersion: 3.0.3
>
> real 0m*17.582s*
> user 0m2.334s
> sys 0m0.470s
>
> On AWS:
> *$ time nodetool version*
> ReleaseVersion: 3.0.3
>
> real 0m*1.084s*
> user 0m1.772s
> sys 0m0.363s
>
> Any way by which its speed can be increased?
>
> Thanks & Regards,
> Bhuvan
>