LevelDB type compaction

[mcasandra <mo...@gmail.com>]

Trying to read through leveldb implementation and had some questions.

With Leveldb is it going to make reads slower since my understanding is it
will create more smaller files and updates could be scattered all over
before compaction? Also, when memtables are flushed, does it create small
files too since memtable size would generally be bigger than 2-4MB?

--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/LevelDB-type-compaction-tp6798334p6798334.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.

Re: LevelDB type compaction

[mcasandra <mo...@gmail.com>]

Can someone please help me understand this a little bit?

--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/LevelDB-type-compaction-tp6798334p6822344.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.

Re: LevelDB type compaction

[Sam Overton <so...@acunu.com>]

On 17 September 2011 00:58, mcasandra <mo...@gmail.com> wrote:

> >
> >> and updates could be scattered all over
> >> before compaction?
> >
> > No, updates to a given row will be still be in a single sstable.
>
> Can you please explain little more? You mean that if Level 1 file contains
> range from 1-100 all the updates would still go in that file?
>

No, sstables are never written to. All updates go to a memtable, which is
then flushed to a Level0 sstable. Compaction then causes promotion of
updates to gradually higher levels.


> The link on leveldb says:
>
> > The compaction picks a file from level L and all overlapping files from
> > the next level L+1
>
> If all updates go in the same sstables then how do overlapping files get
> generated. By overlapping I am assuming it means new or updated value for a
> given key exists in multiple files?
>

Overlapping refers to the range of keys. The key ranges contained within the
files in any given level represent a set of disjoint intervals (when level
>=1). For Level0, the files may contain overlapping ranges.


> Thanks for the explanation
>

Disclaimer: I haven't read the code, this is just my understanding from
reading the docs.


Sam Overton
Acunu | http://www.acunu.com | @acunu

Re: LevelDB type compaction

[mcasandra <mo...@gmail.com>]

> 
>> and updates could be scattered all over
>> before compaction?
> 
> No, updates to a given row will be still be in a single sstable.
> 
> 

Can you please explain little more? You mean that if Level 1 file contains
range from 1-100 all the updates would still go in that file?

The link on leveldb says:



> The compaction picks a file from level L and all overlapping files from
> the next level L+1
> 

If all updates go in the same sstables then how do overlapping files get
generated. By overlapping I am assuming it means new or updated value for a
given key exists in multiple files?

Thanks for the explanation

--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/LevelDB-type-compaction-tp6798334p6802772.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.

Re: LevelDB type compaction

[Jonathan Ellis <jb...@gmail.com>]

On Thu, Sep 15, 2011 at 3:05 PM, mcasandra <mo...@gmail.com> wrote:
> With Leveldb is it going to make reads slower

No.

Qualified: compared to "major compaction" under the tiered strategy,
leveled reads will usually be a little slower for update-heavy loads.
(For insert-mostly workloads compaction doesn't really matter.)  But
major compaction is not practical in production; you want something
that gives consistently good performance, rather than good performance
once a day or once a week and then degrades quickly.

> my understanding is it
> will create more smaller files

Yes.

> and updates could be scattered all over
> before compaction?

No, updates to a given row will be still be in a single sstable.

> Also, when memtables are flushed, does it create small
> files too since memtable size would generally be bigger than 2-4MB?

Level0 (newly flushed) sstables are not size-limited.  This is one of
a handful of differences we have over leveldb itself, which remains a
good overview (http://leveldb.googlecode.com/svn/trunk/doc/impl.html).

-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com