You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Mick Semb Wever <mc...@apache.org> on 2011/10/31 13:05:49 UTC
Re: OOM on CompressionMetadata.readChunkOffsets(..)
On Mon, 2011-10-31 at 09:07 +0100, Mick Semb Wever wrote:
> The read pattern of these rows is always in bulk so the chunk_length
> could have been much higher so to reduce memory usage (my largest
> sstable is 61G).
Isn't CompressionMetadata.readChunkOffsets(..) rather dangerous here?
Given a 60G sstable, even with 64kb chunk_length, to read just that one
sstable requires close to 8G free heap memory...
Especially when the default for cassandra is 4G heap in total.
~mck
--
"Anyone who has attended a computer conference in a fancy hotel can tell
you that a sentence like "You're one of those computer people, aren't
you?" is roughly equivalent to "Look, another amazingly mobile form of
slime mold!" in the mouth of a hotel cocktail waitress." Elizabeth
Zwicky
| http://semb.wever.org | http://sesat.no |
| http://tech.finn.no | Java XSS Filter |
Re: OOM on CompressionMetadata.readChunkOffsets(..)
Posted by Sylvain Lebresne <sy...@datastax.com>.
On Mon, Oct 31, 2011 at 2:58 PM, Sylvain Lebresne <sy...@datastax.com> wrote:
> On Mon, Oct 31, 2011 at 1:10 PM, Mick Semb Wever <mc...@apache.org> wrote:
>> On Mon, 2011-10-31 at 13:05 +0100, Mick Semb Wever wrote:
>>> Given a 60G sstable, even with 64kb chunk_length, to read just that one
>>> sstable requires close to 8G free heap memory...
>>
>> Arg, that calculation was a little off...
>> (a long isn't exactly 8K...)
>>
>> But you get my concern...
>
> Well, with a long being only 8 bytes, that's 8MB of free heap memory. Without
> being negligible, that's not completely crazy to me.
>
> No, the problem is that we create those 8MB for each reads, which *is* crazy
> (the fact that we allocate those 8MB in one block is not very nice for
> the GC either
> but that's another problem).
> Anyway, that's really a bug and I've created CASSANDRA-3427 to fix.
Note that it's only a problem for range queries.
--
Sylvain
>
> --
> Sylvain
>
>>
>> ~mck
>>
>> --
>> "When you say: "I wrote a program that crashed Windows", people just
>> stare at you blankly and say: "Hey, I got those with the system -- for
>> free."" Linus Torvalds
>>
>> | http://semb.wever.org | http://sesat.no |
>> | http://tech.finn.no | Java XSS Filter |
>>
>
Re: OOM on CompressionMetadata.readChunkOffsets(..)
Posted by Sylvain Lebresne <sy...@datastax.com>.
On Mon, Oct 31, 2011 at 1:10 PM, Mick Semb Wever <mc...@apache.org> wrote:
> On Mon, 2011-10-31 at 13:05 +0100, Mick Semb Wever wrote:
>> Given a 60G sstable, even with 64kb chunk_length, to read just that one
>> sstable requires close to 8G free heap memory...
>
> Arg, that calculation was a little off...
> (a long isn't exactly 8K...)
>
> But you get my concern...
Well, with a long being only 8 bytes, that's 8MB of free heap memory. Without
being negligible, that's not completely crazy to me.
No, the problem is that we create those 8MB for each reads, which *is* crazy
(the fact that we allocate those 8MB in one block is not very nice for
the GC either
but that's another problem).
Anyway, that's really a bug and I've created CASSANDRA-3427 to fix.
--
Sylvain
>
> ~mck
>
> --
> "When you say: "I wrote a program that crashed Windows", people just
> stare at you blankly and say: "Hey, I got those with the system -- for
> free."" Linus Torvalds
>
> | http://semb.wever.org | http://sesat.no |
> | http://tech.finn.no | Java XSS Filter |
>
Re: OOM on CompressionMetadata.readChunkOffsets(..)
Posted by Mick Semb Wever <mc...@apache.org>.
On Mon, 2011-10-31 at 13:05 +0100, Mick Semb Wever wrote:
> Given a 60G sstable, even with 64kb chunk_length, to read just that one
> sstable requires close to 8G free heap memory...
Arg, that calculation was a little off...
(a long isn't exactly 8K...)
But you get my concern...
~mck
--
"When you say: "I wrote a program that crashed Windows", people just
stare at you blankly and say: "Hey, I got those with the system -- for
free."" Linus Torvalds
| http://semb.wever.org | http://sesat.no |
| http://tech.finn.no | Java XSS Filter |