You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Aaron Turner <sy...@gmail.com> on 2012/08/11 03:34:58 UTC
quick question about data layout on disk
Curious, but does cassandra store the rowkey along with every
column/value pair on disk (pre-compaction) like Hbase does? If so
(which makes the most sense), I assume that's something that is
optimized during compaction?
--
Aaron Turner
http://synfin.net/ Twitter: @synfinatic
http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
Those who would give up essential Liberty, to purchase a little temporary
Safety, deserve neither Liberty nor Safety.
-- Benjamin Franklin
"carpe diem quam minimum credula postero"
Re: quick question about data layout on disk
Posted by Aaron Turner <sy...@gmail.com>.
Thanks Russell, that's the info I was looking for!
On Sat, Aug 11, 2012 at 11:23 AM, Russell Haering
<ru...@gmail.com> wrote:
> Your update doesn't go directly to an sstable (which are immutable),
> it is first merged to an in-memory table. Eventually the memtable is
> flushed to a new sstable.
>
> See http://wiki.apache.org/cassandra/MemtableSSTable
>
> On Sat, Aug 11, 2012 at 11:03 AM, Aaron Turner <sy...@gmail.com> wrote:
>> So how does that work? An sstable is for a single CF, but it can and
>> likely will have multiple rows. There is no read to write and as I
>> understand it, writes are append operations.
>>
>> So if you have an sstable with say 26 different rows (A-Z) already in
>> it with a bunch of columns and you add a new column to row J, how does
>> Cassandra store the column/value pair on disk in a way to refer to row
>> J without re-writing the row key or some representation of it?
>>
>> Thanks,
>> Aaron
>>
>> On Fri, Aug 10, 2012 at 7:53 PM, Terje Marthinussen
>> <tm...@gmail.com> wrote:
>>> Rowkey is stored only once in any sstable file.
>>>
>>> That is, in the spesial case where you get sstable file per column/value, you are correct, but normally, I guess most of us are storing more per key.
>>>
>>> Regards,
>>> Terje
>>>
>>> On 11 Aug 2012, at 10:34, Aaron Turner <sy...@gmail.com> wrote:
>>>
>>>> Curious, but does cassandra store the rowkey along with every
>>>> column/value pair on disk (pre-compaction) like Hbase does? If so
>>>> (which makes the most sense), I assume that's something that is
>>>> optimized during compaction?
>>>>
>>>>
>>>> --
>>>> Aaron Turner
>>>> http://synfin.net/ Twitter: @synfinatic
>>>> http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
>>>> Those who would give up essential Liberty, to purchase a little temporary
>>>> Safety, deserve neither Liberty nor Safety.
>>>> -- Benjamin Franklin
>>>> "carpe diem quam minimum credula postero"
>>
>>
>>
>> --
>> Aaron Turner
>> http://synfin.net/ Twitter: @synfinatic
>> http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
>> Those who would give up essential Liberty, to purchase a little temporary
>> Safety, deserve neither Liberty nor Safety.
>> -- Benjamin Franklin
>> "carpe diem quam minimum credula postero"
--
Aaron Turner
http://synfin.net/ Twitter: @synfinatic
http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
Those who would give up essential Liberty, to purchase a little temporary
Safety, deserve neither Liberty nor Safety.
-- Benjamin Franklin
"carpe diem quam minimum credula postero"
Re: quick question about data layout on disk
Posted by Russell Haering <ru...@gmail.com>.
Your update doesn't go directly to an sstable (which are immutable),
it is first merged to an in-memory table. Eventually the memtable is
flushed to a new sstable.
See http://wiki.apache.org/cassandra/MemtableSSTable
On Sat, Aug 11, 2012 at 11:03 AM, Aaron Turner <sy...@gmail.com> wrote:
> So how does that work? An sstable is for a single CF, but it can and
> likely will have multiple rows. There is no read to write and as I
> understand it, writes are append operations.
>
> So if you have an sstable with say 26 different rows (A-Z) already in
> it with a bunch of columns and you add a new column to row J, how does
> Cassandra store the column/value pair on disk in a way to refer to row
> J without re-writing the row key or some representation of it?
>
> Thanks,
> Aaron
>
> On Fri, Aug 10, 2012 at 7:53 PM, Terje Marthinussen
> <tm...@gmail.com> wrote:
>> Rowkey is stored only once in any sstable file.
>>
>> That is, in the spesial case where you get sstable file per column/value, you are correct, but normally, I guess most of us are storing more per key.
>>
>> Regards,
>> Terje
>>
>> On 11 Aug 2012, at 10:34, Aaron Turner <sy...@gmail.com> wrote:
>>
>>> Curious, but does cassandra store the rowkey along with every
>>> column/value pair on disk (pre-compaction) like Hbase does? If so
>>> (which makes the most sense), I assume that's something that is
>>> optimized during compaction?
>>>
>>>
>>> --
>>> Aaron Turner
>>> http://synfin.net/ Twitter: @synfinatic
>>> http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
>>> Those who would give up essential Liberty, to purchase a little temporary
>>> Safety, deserve neither Liberty nor Safety.
>>> -- Benjamin Franklin
>>> "carpe diem quam minimum credula postero"
>
>
>
> --
> Aaron Turner
> http://synfin.net/ Twitter: @synfinatic
> http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
> Those who would give up essential Liberty, to purchase a little temporary
> Safety, deserve neither Liberty nor Safety.
> -- Benjamin Franklin
> "carpe diem quam minimum credula postero"
Re: quick question about data layout on disk
Posted by Edward Capriolo <ed...@gmail.com>.
Aaron,
I have not deep dived the data files in a while but this is how I understand it.
http://wiki.apache.org/cassandra/ArchitectureSSTable
There is no need to store the row key each time with the column.
RowKey to columns is a one to many relationship. This would be a
diagram of a physical file:
Hbase does it like this i guess (Or it used to I am not up on the news):
rowkey1,column1,value1
rowkey1,column2,value2
First, I believe they repeat the row key for each column, which is not
a huge deal because you should always use compression but it is a bit
wasteful especially for a non-compressed table.
I know this has some impact on very wide rows because a single rowkey
must fit inside this structure of an hfile. ? (again its been a while)
But to get back to your question. In cassandra:
sstable1
rowkey1: numberof columns 26
(column1,value1,ts1)
....
(column26,value26,ts26)
sstable2
rowkey1: numberof columns 1
(column1,value1,ts2)
The rowkey appears once in a given sstable if the row has 1 or more
columns in the sstable.
On the read path Cassandra searches all sstables find all the columns
for a row (bloom filters and other criteria eliminate some sstables
from read path). It then merges the row factoring in tombstones and
the last update win rules for a column.
On Sat, Aug 11, 2012 at 2:03 PM, Aaron Turner <sy...@gmail.com> wrote:
> So how does that work? An sstable is for a single CF, but it can and
> likely will have multiple rows. There is no read to write and as I
> understand it, writes are append operations.
>
> So if you have an sstable with say 26 different rows (A-Z) already in
> it with a bunch of columns and you add a new column to row J, how does
> Cassandra store the column/value pair on disk in a way to refer to row
> J without re-writing the row key or some representation of it?
>
> Thanks,
> Aaron
>
> On Fri, Aug 10, 2012 at 7:53 PM, Terje Marthinussen
> <tm...@gmail.com> wrote:
>> Rowkey is stored only once in any sstable file.
>>
>> That is, in the spesial case where you get sstable file per column/value, you are correct, but normally, I guess most of us are storing more per key.
>>
>> Regards,
>> Terje
>>
>> On 11 Aug 2012, at 10:34, Aaron Turner <sy...@gmail.com> wrote:
>>
>>> Curious, but does cassandra store the rowkey along with every
>>> column/value pair on disk (pre-compaction) like Hbase does? If so
>>> (which makes the most sense), I assume that's something that is
>>> optimized during compaction?
>>>
>>>
>>> --
>>> Aaron Turner
>>> http://synfin.net/ Twitter: @synfinatic
>>> http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
>>> Those who would give up essential Liberty, to purchase a little temporary
>>> Safety, deserve neither Liberty nor Safety.
>>> -- Benjamin Franklin
>>> "carpe diem quam minimum credula postero"
>
>
>
> --
> Aaron Turner
> http://synfin.net/ Twitter: @synfinatic
> http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
> Those who would give up essential Liberty, to purchase a little temporary
> Safety, deserve neither Liberty nor Safety.
> -- Benjamin Franklin
> "carpe diem quam minimum credula postero"
Re: quick question about data layout on disk
Posted by Aaron Turner <sy...@gmail.com>.
So how does that work? An sstable is for a single CF, but it can and
likely will have multiple rows. There is no read to write and as I
understand it, writes are append operations.
So if you have an sstable with say 26 different rows (A-Z) already in
it with a bunch of columns and you add a new column to row J, how does
Cassandra store the column/value pair on disk in a way to refer to row
J without re-writing the row key or some representation of it?
Thanks,
Aaron
On Fri, Aug 10, 2012 at 7:53 PM, Terje Marthinussen
<tm...@gmail.com> wrote:
> Rowkey is stored only once in any sstable file.
>
> That is, in the spesial case where you get sstable file per column/value, you are correct, but normally, I guess most of us are storing more per key.
>
> Regards,
> Terje
>
> On 11 Aug 2012, at 10:34, Aaron Turner <sy...@gmail.com> wrote:
>
>> Curious, but does cassandra store the rowkey along with every
>> column/value pair on disk (pre-compaction) like Hbase does? If so
>> (which makes the most sense), I assume that's something that is
>> optimized during compaction?
>>
>>
>> --
>> Aaron Turner
>> http://synfin.net/ Twitter: @synfinatic
>> http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
>> Those who would give up essential Liberty, to purchase a little temporary
>> Safety, deserve neither Liberty nor Safety.
>> -- Benjamin Franklin
>> "carpe diem quam minimum credula postero"
--
Aaron Turner
http://synfin.net/ Twitter: @synfinatic
http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
Those who would give up essential Liberty, to purchase a little temporary
Safety, deserve neither Liberty nor Safety.
-- Benjamin Franklin
"carpe diem quam minimum credula postero"
Re: quick question about data layout on disk
Posted by Terje Marthinussen <tm...@gmail.com>.
Rowkey is stored only once in any sstable file.
That is, in the spesial case where you get sstable file per column/value, you are correct, but normally, I guess most of us are storing more per key.
Regards,
Terje
On 11 Aug 2012, at 10:34, Aaron Turner <sy...@gmail.com> wrote:
> Curious, but does cassandra store the rowkey along with every
> column/value pair on disk (pre-compaction) like Hbase does? If so
> (which makes the most sense), I assume that's something that is
> optimized during compaction?
>
>
> --
> Aaron Turner
> http://synfin.net/ Twitter: @synfinatic
> http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
> Those who would give up essential Liberty, to purchase a little temporary
> Safety, deserve neither Liberty nor Safety.
> -- Benjamin Franklin
> "carpe diem quam minimum credula postero"