You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Cyril Scetbon <cy...@free.fr> on 2016/05/31 15:59:05 UTC
Computing ranges takes time
Hi C* developers,
Digging in the code because of a time issue during a repair full on our ~ 60 nodes cluster, I've been able to see that this stage can be significant (up to 60 percent of) :
https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/service/StorageService.java#L2983-L2997
It's merely caused by the fact that https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/service/ActiveRepairService.java#L189 calls `ss.getLocalRanges(keyspaceName)` everytime and that it takes more than 99% of the time. This call takes 600ms when there is no load on the cluster and more if there is. So for 10k ranges, you can imagine that it takes at least 1.5 hours just to compute ranges. Don't you think that caching this call would make sense ?
--
Cyril SCETBON
Re: Computing ranges takes time
Posted by Cyril Scetbon <cy...@free.fr>.
Ticket created at https://issues.apache.org/jira/browse/CASSANDRA-11933
Thanks
> On May 31, 2016, at 13:54, Paulo Motta <pa...@gmail.com> wrote:
>
> Good catch! It definitely makes sense to cache this call for a single
> repair job as it calls ReplicationStrategy.getAddressRanges underneath
> which can get pretty inefficient (
> https://github.com/apache/cassandra/blob/3dcbe90e02440e6ee534f643c7603d50ca08482b/src/java/org/apache/cassandra/locator/AbstractReplicationStrategy.java#L170
> ).
>
> Would you mind creating a ticket and submitting a patch?
>
> Thanks!
>
> 2016-05-31 12:59 GMT-03:00 Cyril Scetbon <cy...@free.fr>:
>
>> Hi C* developers,
>>
>> Digging in the code because of a time issue during a repair full on our ~
>> 60 nodes cluster, I've been able to see that this stage can be significant
>> (up to 60 percent of) :
>>
>>
>> https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/service/StorageService.java#L2983-L2997
>>
>> It's merely caused by the fact that
>> https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/service/ActiveRepairService.java#L189
>> calls `ss.getLocalRanges(keyspaceName)` everytime and that it takes more
>> than 99% of the time. This call takes 600ms when there is no load on the
>> cluster and more if there is. So for 10k ranges, you can imagine that it
>> takes at least 1.5 hours just to compute ranges. Don't you think that
>> caching this call would make sense ?
>>
>> --
>> Cyril SCETBON
>>
>>
Re: Computing ranges takes time
Posted by Paulo Motta <pa...@gmail.com>.
Good catch! It definitely makes sense to cache this call for a single
repair job as it calls ReplicationStrategy.getAddressRanges underneath
which can get pretty inefficient (
https://github.com/apache/cassandra/blob/3dcbe90e02440e6ee534f643c7603d50ca08482b/src/java/org/apache/cassandra/locator/AbstractReplicationStrategy.java#L170
).
Would you mind creating a ticket and submitting a patch?
Thanks!
2016-05-31 12:59 GMT-03:00 Cyril Scetbon <cy...@free.fr>:
> Hi C* developers,
>
> Digging in the code because of a time issue during a repair full on our ~
> 60 nodes cluster, I've been able to see that this stage can be significant
> (up to 60 percent of) :
>
>
> https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/service/StorageService.java#L2983-L2997
>
> It's merely caused by the fact that
> https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/service/ActiveRepairService.java#L189
> calls `ss.getLocalRanges(keyspaceName)` everytime and that it takes more
> than 99% of the time. This call takes 600ms when there is no load on the
> cluster and more if there is. So for 10k ranges, you can imagine that it
> takes at least 1.5 hours just to compute ranges. Don't you think that
> caching this call would make sense ?
>
> --
> Cyril SCETBON
>
>