Re: 1.2 tuning

[Darren Smythe <da...@gmail.com>]

If the amount of remaining time for compaction keeps going up, does this
point to an overloaded node or an un-tuned node?


On Thu, May 30, 2013 at 3:10 PM, Robert Coli <rc...@eventbrite.com> wrote:

> On Wed, May 29, 2013 at 2:38 PM, Darren Smythe <da...@gmail.com>
> wrote:
> > Were using the latest JNA and separate ephemeral drives for commit log
> and
> > data directories.
>
> (as a note..)
>
> Per nickmbailey, testing shows that there is little/no benefit to
> separating commit log and data dirs on virtualized disk (or SSD),
> because the win from this practice comes when the head doesn't move
> between appends to the commit log. Because   the head must be assumed
> to always be moving on shared disk (and because there is no head to
> move on SSD), you'd be better off with a one-disk-larger ephemeral
> stripe for both data and commit log.
>
> =Rob
>

Re: 1.2 tuning

[Darren Smythe <da...@gmail.com>]

Hi-

On Mon, Jun 3, 2013 at 10:36 AM, Robert Coli <rc...@eventbrite.com> wrote:

> On Mon, Jun 3, 2013 at 8:54 AM, Darren Smythe <da...@gmail.com>
> wrote:
> > Is "setting live ratio to minimum of 1.0 instead of X" supposed to be
> rare?
> > Because were getting it fairly consistently.
>
> Do you have working JNA? If so, my understanding is that message
> should be relatively rare..
>


We recently tried a 8 xlarge node cluster instead of just 4 xlarge
instances, keeping the number of clients the same as before. But were still
seeing a lot of "setting live ratio" and "flush up to the two largest
memtables" messages alot. The compaction queues still seem congested also.

Theere's also alot of dropped MUTATION messages --- 13,000,000 Completed in
Mutation Stage, 100,000 Dropped

THe inserts are now about 11k/second, up slightly from 10k/second with 4
nodes.

Another missing piece may be that each column has a 1-2k blob as a value.
The value is binary now, but were moving to json later.

TIA
--Darren


>
> ./src/java/org/apache/cassandra/db/Memtable.java
> "
>               double newRatio = (double) deepSize / currentSize.get();
>
>                     if (newRatio < MIN_SANE_LIVE_RATIO)
>                     {
>                         logger.warn("setting live ratio to minimum of
> {} instead of {}", MIN_SANE_LIVE_RATIO, newRatio);
>                         newRatio = MIN_SANE_LIVE_RATIO;
>                     }
> "
>
> =Rob
>

Re: 1.2 tuning

[Robert Coli <rc...@eventbrite.com>]

On Mon, Jun 3, 2013 at 8:54 AM, Darren Smythe <da...@gmail.com> wrote:
> Is "setting live ratio to minimum of 1.0 instead of X" supposed to be rare?
> Because were getting it fairly consistently.

Do you have working JNA? If so, my understanding is that message
should be relatively rare..

./src/java/org/apache/cassandra/db/Memtable.java
"
              double newRatio = (double) deepSize / currentSize.get();

                    if (newRatio < MIN_SANE_LIVE_RATIO)
                    {
                        logger.warn("setting live ratio to minimum of
{} instead of {}", MIN_SANE_LIVE_RATIO, newRatio);
                        newRatio = MIN_SANE_LIVE_RATIO;
                    }
"

=Rob

Re: 1.2 tuning

[Darren Smythe <da...@gmail.com>]

Is "setting live ratio to minimum of 1.0 instead of X" supposed to be rare?
Because were getting it fairly consistently.


On Sat, Jun 1, 2013 at 8:58 PM, Darren Smythe <da...@gmail.com> wrote:

> If the amount of remaining time for compaction keeps going up, does this
> point to an overloaded node or an un-tuned node?
>
>
> On Thu, May 30, 2013 at 3:10 PM, Robert Coli <rc...@eventbrite.com> wrote:
>
>> On Wed, May 29, 2013 at 2:38 PM, Darren Smythe <da...@gmail.com>
>> wrote:
>> > Were using the latest JNA and separate ephemeral drives for commit log
>> and
>> > data directories.
>>
>> (as a note..)
>>
>> Per nickmbailey, testing shows that there is little/no benefit to
>> separating commit log and data dirs on virtualized disk (or SSD),
>> because the win from this practice comes when the head doesn't move
>> between appends to the commit log. Because   the head must be assumed
>> to always be moving on shared disk (and because there is no head to
>> move on SSD), you'd be better off with a one-disk-larger ephemeral
>> stripe for both data and commit log.
>>
>> =Rob
>>
>
>