You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Paulo Ricardo Motta Gomes <pa...@chaordicsystems.com> on 2014/11/10 15:18:00 UTC
Increase in dropped mutations after major upgrade from 1.2.18 to 2.0.10
Hey,
We've seen a considerable increase in the number of dropped mutations after
a major upgrade from 1.2.18 to 2.0.10. I initially thought it was due to
the extra load incurred by upgradesstables, but the dropped mutations
continue even after all sstables are upgraded.
Additional info: Overall (read, write and range) latency improved with the
upgrade, which is great, but I don't understand why dropped mutations has
increased. I/O and CPU load is pretty much the same, number of completed
tasks is the only metric that increased together with dropped mutations.
I also noticed that the number of "all time blocked" FlushWriter operations
is about 5% of completed operations, don't know if this is related, but in
case it helps out...
Anyone has a clue on what could that be? Or what should we monitor to find
out? Any help or JIRA pointers would be kindly appreciated.
Cheers,
--
*Paulo Motta*
Chaordic | *Platform*
*www.chaordic.com.br <http://www.chaordic.com.br/>*
+55 48 3232.3200
Re: Increase in dropped mutations after major upgrade from 1.2.18 to 2.0.10
Posted by Paulo Ricardo Motta Gomes <pa...@chaordicsystems.com>.
On Mon, Nov 10, 2014 at 12:46 PM, Duncan Sands <du...@gmail.com>
wrote:
> Hi Paulo,
>
> On 10/11/14 15:18, Paulo Ricardo Motta Gomes wrote:
>
>> Hey,
>>
>> We've seen a considerable increase in the number of dropped mutations
>> after a
>> major upgrade from 1.2.18 to 2.0.10. I initially thought it was due to
>> the extra
>> load incurred by upgradesstables, but the dropped mutations continue even
>> after
>> all sstables are upgraded.
>>
>
> are the clocks on all your nodes synchronized with each other?
>
> Ciao, Duncan.
>
Yes, the servers are synchronized via NTP.
Cheers!
>
>
>> Additional info: Overall (read, write and range) latency improved with the
>> upgrade, which is great, but I don't understand why dropped mutations has
>> increased. I/O and CPU load is pretty much the same, number of completed
>> tasks
>> is the only metric that increased together with dropped mutations.
>>
>> I also noticed that the number of "all time blocked" FlushWriter
>> operations is
>> about 5% of completed operations, don't know if this is related, but in
>> case it
>> helps out...
>>
>> Anyone has a clue on what could that be? Or what should we monitor to
>> find out?
>> Any help or JIRA pointers would be kindly appreciated.
>>
>> Cheers,
>>
>> --
>> *Paulo Motta*
>>
>> Chaordic | /Platform/
>> _www.chaordic.com.br <http://www.chaordic.com.br/>_
>> +55 48 3232.3200
>>
>
>
--
*Paulo Motta*
Chaordic | *Platform*
*www.chaordic.com.br <http://www.chaordic.com.br/>*
+55 48 3232.3200
Re: Increase in dropped mutations after major upgrade from 1.2.18
to 2.0.10
Posted by Duncan Sands <du...@gmail.com>.
Hi Paulo,
On 10/11/14 15:18, Paulo Ricardo Motta Gomes wrote:
> Hey,
>
> We've seen a considerable increase in the number of dropped mutations after a
> major upgrade from 1.2.18 to 2.0.10. I initially thought it was due to the extra
> load incurred by upgradesstables, but the dropped mutations continue even after
> all sstables are upgraded.
are the clocks on all your nodes synchronized with each other?
Ciao, Duncan.
>
> Additional info: Overall (read, write and range) latency improved with the
> upgrade, which is great, but I don't understand why dropped mutations has
> increased. I/O and CPU load is pretty much the same, number of completed tasks
> is the only metric that increased together with dropped mutations.
>
> I also noticed that the number of "all time blocked" FlushWriter operations is
> about 5% of completed operations, don't know if this is related, but in case it
> helps out...
>
> Anyone has a clue on what could that be? Or what should we monitor to find out?
> Any help or JIRA pointers would be kindly appreciated.
>
> Cheers,
>
> --
> *Paulo Motta*
>
> Chaordic | /Platform/
> _www.chaordic.com.br <http://www.chaordic.com.br/>_
> +55 48 3232.3200