You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Kevin Burton <bu...@spinn3r.com> on 2014/07/02 02:54:42 UTC
Fetching ONE cell with a row cache hit takes 1 second on an idle box?
I'm really perplexed on this one.. I think this must be a bug or some
misconfiguration somewhere.
I'm fetching ONE row which is one cell in my table.
The row is in the row cache, sitting in memory.
SELECT sequence from content where bucket=98 AND sequence =
1403481494000005742;
And look at the trace below… it's taking 1000ms to go to the row cache?
That seems insane!
Something must be broken somewhere.. Any advice in what I should be looking
into?
The VM has plenty of memory so I don't think it's GC.
activity
| timestamp | source | source_elapsed
------------------------------------------------------------------------------------------------------+--------------+-------------+----------------
execute_cql3_query | 19:48:48,559 | 10.24.23.94 | 0
Parsing SELECT sequence from content where bucket=98 AND sequence =
1403481494000005742 LIMIT 10000; | 19:48:48,559 | 10.24.23.94 |
63
Preparing statement | 19:48:48,559 | 10.24.23.94 | 153
Row cache hit | 19:48:49,706 | 10.24.23.94 | 1147108
Read 1
live and 0 tombstoned cells | 19:48:49,706 | 10.24.23.94 | 1147236
Request complete | 19:48:49,706 | 10.24.23.94 | 1147412
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: Fetching ONE cell with a row cache hit takes 1 second on an idle box?
Posted by Colin <co...@gmail.com>.
Rowcache is typically turned off because it is only useful in very specific situations-the row(s) need to fit in memory. Also, the access patterns have to fit.
If all the rows you're accessing can fit, Rowcache is a great thing. Otherwise, not so great.
--
Colin
320-221-9531
> On Jul 1, 2014, at 10:40 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
> WOW.. so based on your advice, and a test, I disabled the row cache for the table.
>
> The query was instantly 20x faster.
>
> so this is definitely an anti-pattern.. I suspect cassandra just tries to read in they entire physical row into memory and since my physical row is rather big.. ha. Well that wasn't very fun :)
>
> BIG win though ;)
>
>
>> On Tue, Jul 1, 2014 at 6:52 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>> A work around for this, is the VFS page cache.. basically, disabling compression, and then allowing the VFS page cache to keep your data in memory.
>>
>> The only downside is the per column overhead. But if you can store everything in a 'blob' which is optionally compressed, you're generally going to be ok.
>>
>> Kevin
>>
>>
>>> On Tue, Jul 1, 2014 at 6:50 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>>> so.. caching the *queries* ?
>>>
>>> it seems like a better mechanism would be to cache the actually logical row…, not the physical row.
>>>
>>> Query caches just don't work in production, If you re-word your query, or structure it a different way, you get a miss…
>>>
>>> In my experience.. query caches have a 0% hit rate.
>>>
>>>
>>>> On Tue, Jul 1, 2014 at 6:40 PM, Robert Coli <rc...@eventbrite.com> wrote:
>>>>> On Tue, Jul 1, 2014 at 6:06 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>>>>> you know.. one thing I failed to mention.. .is that this is going into a "bucket" and while it's a logical row, the physical row is like 500MB … according to compaction logs.
>>>>>
>>>>> is the ENTIRE physical row going into the cache as one unit? That's definitely going to be a problem in this model. 500MB is a big atomic unit.
>>>>
>>>> Yes, the row cache is a row cache. It caches what the storage engine calls rows, which CQL calls "partitions." [1] Rows have to be assembled from all of their row fragments in Memtables/SSTables.
>>>>
>>>> This is a big part of why the "off-heap" row cache's behavior of invalidation on write is so bad for its overall performance. Updating a single column in your 500MB row invalidates it and forces you to assemble the entire 500MB row from disk.
>>>>
>>>> The only valid use case for the current off-heap row cache seems to be : very small, very uniform in size, very hot, and very rarely modified.
>>>>
>>>> https://issues.apache.org/jira/browse/CASSANDRA-5357
>>>>
>>>> Is the ticket for replacing the row cache and its unexpected characteristics with something more like an actual query cache.
>>>>
>>>>> also.. I assume it's having to do a binary search within the physical row ?
>>>>
>>>> Since the column level bloom filter's removal in 1.2, the only way it can get to specific columns is via the index.
>>>>
>>>> =Rob
>>>> [1] https://issues.apache.org/jira/browse/CASSANDRA-6632
>>>
>>>
>>>
>>> --
>>> Founder/CEO Spinn3r.com
>>> Location: San Francisco, CA
>>> blog: http://burtonator.wordpress.com
>>> … or check out my Google+ profile
>>>
>>
>>
>>
>> --
>> Founder/CEO Spinn3r.com
>> Location: San Francisco, CA
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>>
>
>
>
> --
> Founder/CEO Spinn3r.com
> Location: San Francisco, CA
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
>
Re: Fetching ONE cell with a row cache hit takes 1 second on an idle box?
Posted by Kevin Burton <bu...@spinn3r.com>.
WOW.. so based on your advice, and a test, I disabled the row cache for the
table.
The query was instantly 20x faster.
so this is definitely an anti-pattern.. I suspect cassandra just tries to
read in they entire physical row into memory and since my physical row is
rather big.. ha. Well that wasn't very fun :)
BIG win though ;)
On Tue, Jul 1, 2014 at 6:52 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> A work around for this, is the VFS page cache.. basically, disabling
> compression, and then allowing the VFS page cache to keep your data in
> memory.
>
> The only downside is the per column overhead. But if you can store
> everything in a 'blob' which is optionally compressed, you're generally
> going to be ok.
>
> Kevin
>
>
> On Tue, Jul 1, 2014 at 6:50 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
>> so.. caching the *queries* ?
>>
>> it seems like a better mechanism would be to cache the actually logical
>> row…, not the physical row.
>>
>> Query caches just don't work in production, If you re-word your query,
>> or structure it a different way, you get a miss…
>>
>> In my experience.. query caches have a 0% hit rate.
>>
>>
>> On Tue, Jul 1, 2014 at 6:40 PM, Robert Coli <rc...@eventbrite.com> wrote:
>>
>>> On Tue, Jul 1, 2014 at 6:06 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>>>
>>>> you know.. one thing I failed to mention.. .is that this is going into
>>>> a "bucket" and while it's a logical row, the physical row is like 500MB …
>>>> according to compaction logs.
>>>>
>>>> is the ENTIRE physical row going into the cache as one unit? That's
>>>> definitely going to be a problem in this model. 500MB is a big atomic unit.
>>>>
>>>
>>> Yes, the row cache is a row cache. It caches what the storage engine
>>> calls rows, which CQL calls "partitions." [1] Rows have to be assembled
>>> from all of their row fragments in Memtables/SSTables.
>>>
>>> This is a big part of why the "off-heap" row cache's behavior of
>>> invalidation on write is so bad for its overall performance. Updating a
>>> single column in your 500MB row invalidates it and forces you to assemble
>>> the entire 500MB row from disk.
>>>
>>> The only valid use case for the current off-heap row cache seems to be :
>>> very small, very uniform in size, very hot, and very rarely modified.
>>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-5357
>>>
>>> Is the ticket for replacing the row cache and its unexpected
>>> characteristics with something more like an actual query cache.
>>>
>>> also.. I assume it's having to do a binary search within the physical
>>>> row ?
>>>
>>>
>>> Since the column level bloom filter's removal in 1.2, the only way it
>>> can get to specific columns is via the index.
>>>
>>> =Rob
>>> [1] https://issues.apache.org/jira/browse/CASSANDRA-6632
>>>
>>
>>
>>
>> --
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> <https://plus.google.com/102718274791889610666/posts>
>> <http://spinn3r.com>
>>
>>
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: Fetching ONE cell with a row cache hit takes 1 second on an idle box?
Posted by Kevin Burton <bu...@spinn3r.com>.
A work around for this, is the VFS page cache.. basically, disabling
compression, and then allowing the VFS page cache to keep your data in
memory.
The only downside is the per column overhead. But if you can store
everything in a 'blob' which is optionally compressed, you're generally
going to be ok.
Kevin
On Tue, Jul 1, 2014 at 6:50 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> so.. caching the *queries* ?
>
> it seems like a better mechanism would be to cache the actually logical
> row…, not the physical row.
>
> Query caches just don't work in production, If you re-word your query, or
> structure it a different way, you get a miss…
>
> In my experience.. query caches have a 0% hit rate.
>
>
> On Tue, Jul 1, 2014 at 6:40 PM, Robert Coli <rc...@eventbrite.com> wrote:
>
>> On Tue, Jul 1, 2014 at 6:06 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>>
>>> you know.. one thing I failed to mention.. .is that this is going into a
>>> "bucket" and while it's a logical row, the physical row is like 500MB …
>>> according to compaction logs.
>>>
>>> is the ENTIRE physical row going into the cache as one unit? That's
>>> definitely going to be a problem in this model. 500MB is a big atomic unit.
>>>
>>
>> Yes, the row cache is a row cache. It caches what the storage engine
>> calls rows, which CQL calls "partitions." [1] Rows have to be assembled
>> from all of their row fragments in Memtables/SSTables.
>>
>> This is a big part of why the "off-heap" row cache's behavior of
>> invalidation on write is so bad for its overall performance. Updating a
>> single column in your 500MB row invalidates it and forces you to assemble
>> the entire 500MB row from disk.
>>
>> The only valid use case for the current off-heap row cache seems to be :
>> very small, very uniform in size, very hot, and very rarely modified.
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-5357
>>
>> Is the ticket for replacing the row cache and its unexpected
>> characteristics with something more like an actual query cache.
>>
>> also.. I assume it's having to do a binary search within the physical row
>>> ?
>>
>>
>> Since the column level bloom filter's removal in 1.2, the only way it can
>> get to specific columns is via the index.
>>
>> =Rob
>> [1] https://issues.apache.org/jira/browse/CASSANDRA-6632
>>
>
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: Fetching ONE cell with a row cache hit takes 1 second on an idle box?
Posted by Kevin Burton <bu...@spinn3r.com>.
so.. caching the *queries* ?
it seems like a better mechanism would be to cache the actually logical
row…, not the physical row.
Query caches just don't work in production, If you re-word your query, or
structure it a different way, you get a miss…
In my experience.. query caches have a 0% hit rate.
On Tue, Jul 1, 2014 at 6:40 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Tue, Jul 1, 2014 at 6:06 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
>> you know.. one thing I failed to mention.. .is that this is going into a
>> "bucket" and while it's a logical row, the physical row is like 500MB …
>> according to compaction logs.
>>
>> is the ENTIRE physical row going into the cache as one unit? That's
>> definitely going to be a problem in this model. 500MB is a big atomic unit.
>>
>
> Yes, the row cache is a row cache. It caches what the storage engine calls
> rows, which CQL calls "partitions." [1] Rows have to be assembled from all
> of their row fragments in Memtables/SSTables.
>
> This is a big part of why the "off-heap" row cache's behavior of
> invalidation on write is so bad for its overall performance. Updating a
> single column in your 500MB row invalidates it and forces you to assemble
> the entire 500MB row from disk.
>
> The only valid use case for the current off-heap row cache seems to be :
> very small, very uniform in size, very hot, and very rarely modified.
>
> https://issues.apache.org/jira/browse/CASSANDRA-5357
>
> Is the ticket for replacing the row cache and its unexpected
> characteristics with something more like an actual query cache.
>
> also.. I assume it's having to do a binary search within the physical row
>> ?
>
>
> Since the column level bloom filter's removal in 1.2, the only way it can
> get to specific columns is via the index.
>
> =Rob
> [1] https://issues.apache.org/jira/browse/CASSANDRA-6632
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: Fetching ONE cell with a row cache hit takes 1 second on an idle box?
Posted by Robert Coli <rc...@eventbrite.com>.
On Tue, Jul 1, 2014 at 6:06 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> you know.. one thing I failed to mention.. .is that this is going into a
> "bucket" and while it's a logical row, the physical row is like 500MB …
> according to compaction logs.
>
> is the ENTIRE physical row going into the cache as one unit? That's
> definitely going to be a problem in this model. 500MB is a big atomic unit.
>
Yes, the row cache is a row cache. It caches what the storage engine calls
rows, which CQL calls "partitions." [1] Rows have to be assembled from all
of their row fragments in Memtables/SSTables.
This is a big part of why the "off-heap" row cache's behavior of
invalidation on write is so bad for its overall performance. Updating a
single column in your 500MB row invalidates it and forces you to assemble
the entire 500MB row from disk.
The only valid use case for the current off-heap row cache seems to be :
very small, very uniform in size, very hot, and very rarely modified.
https://issues.apache.org/jira/browse/CASSANDRA-5357
Is the ticket for replacing the row cache and its unexpected
characteristics with something more like an actual query cache.
also.. I assume it's having to do a binary search within the physical row ?
Since the column level bloom filter's removal in 1.2, the only way it can
get to specific columns is via the index.
=Rob
[1] https://issues.apache.org/jira/browse/CASSANDRA-6632
Re: Fetching ONE cell with a row cache hit takes 1 second on an idle box?
Posted by Kevin Burton <bu...@spinn3r.com>.
you know.. one thing I failed to mention.. .is that this is going into a
"bucket" and while it's a logical row, the physical row is like 500MB …
according to compaction logs.
is the ENTIRE physical row going into the cache as one unit? That's
definitely going to be a problem in this model. 500MB is a big atomic unit.
also.. I assume it's having to do a binary search within the physical row ?
On Tue, Jul 1, 2014 at 5:54 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> I'm really perplexed on this one.. I think this must be a bug or some
> misconfiguration somewhere.
>
> I'm fetching ONE row which is one cell in my table.
>
> The row is in the row cache, sitting in memory.
>
> SELECT sequence from content where bucket=98 AND sequence =
> 1403481494000005742;
>
> And look at the trace below… it's taking 1000ms to go to the row cache?
> That seems insane!
>
> Something must be broken somewhere.. Any advice in what I should be
> looking into?
>
> The VM has plenty of memory so I don't think it's GC.
>
> activity
> | timestamp | source | source_elapsed
>
> ------------------------------------------------------------------------------------------------------+--------------+-------------+----------------
>
> execute_cql3_query | 19:48:48,559 | 10.24.23.94 | 0
> Parsing SELECT sequence from content where bucket=98 AND sequence =
> 1403481494000005742 LIMIT 10000; | 19:48:48,559 | 10.24.23.94 |
> 63
>
> Preparing statement | 19:48:48,559 | 10.24.23.94 | 153
>
> Row cache hit | 19:48:49,706 | 10.24.23.94 | 1147108
> Read 1
> live and 0 tombstoned cells | 19:48:49,706 | 10.24.23.94 | 1147236
>
> Request complete | 19:48:49,706 | 10.24.23.94 | 1147412
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>