Re: Running repair negatively impacts read performance?

[Charles Brophy <cb...@zulily.com>]

I should have done more research before asking the question. I mean real
research, too :)

I did a before repair, after repair, and after scrub cfstat. On a hunch I
also did a before/after repair but with no scrub - instead I left the
cluster alone for the length of time that a scrub normally takes (which can
be hours on our dataset). It turns out that in all probability it's just a
waiting game. The bloom filter stats were relatively identical at the end
of the equivalent time period, as was the query performance. I guess I just
needed to wait longer for the streamed files to "settle" or some such.

Thanks
Charles

On Fri, Sep 28, 2012 at 7:20 AM, Charles Brophy <cb...@zulily.com> wrote:

> Odd indeed.
>
> 1) It is observable after the compactions are through and the system has
> "settled"
> 2) We're using SizeTiered strategy
> 3) CentOS 6 & Oracle JVM 1.6.31
>
> I'll do a repair and get some before/after stats to answer your remaining
> questions.
>
> Thanks Aaron
>
> On Wed, Sep 26, 2012 at 2:51 PM, aaron morton <aa...@thelastpickle.com>wrote:
>
>> Sounds very odd.
>>
>> Is read performance degrading _after_ repair and compactions that
>> normally result have completed ?
>> What Compaction Strategy ?
>> What OS and JVM ?
>>
>> What are are the bloom filter false positive stats from cf stats ?
>>
>> Do you have some read latency numbers from cfstats ?
>> Also, could you take a look at cfhistograms  ?
>>
>> Cheers
>>
>>
>>   -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 26/09/2012, at 3:05 AM, Charles Brophy <cb...@zulily.com> wrote:
>>
>> Hey guys,
>>
>> I've begun to notice that read operations take a performance nose-dive
>> after a standard (full) repair of a fairly large column family: ~11 million
>> records. Interestingly, I've then noticed that read performance returns to
>> normal after a full scrub of the column family. Is it possible that the
>> repair operation is not correctly establishing the bloom filter afterwards?
>> I've noticed an interesting note of the scrub operation is that it will
>> "rebuild sstables with correct bloom filters" which is what is leading me
>> to this conclusion. Does this make sense?
>>
>> I'm using 1.1.3 and Oracle JDK 1.6.31
>> The column family is a stanard type and I've noticed this exact behavior
>> regardless of the key/column/value serializers in use.
>>
>> Charles
>>
>>
>>
>