You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jon Haddad <jo...@jonhaddad.com> on 2018/02/20 21:26:02 UTC
Re: Memtable flush -> SSTable: customizable or same for all
compaction strategies?
The file format is independent from compaction. A compaction strategy only selects sstables to be compacted, that’s it’s only job. It could have side effects, like generating other files, but any decent compaction strategy will account for the fact that those other files don’t exist.
I wrote a blog post a few months ago going over some of the nuance of compaction you mind find informative: http://thelastpickle.com/blog/2017/03/16/compaction-nuance.html <http://thelastpickle.com/blog/2017/03/16/compaction-nuance.html>
This is also the wrong mailing list, please direct future user questions to the user list. The dev list is for development of Cassandra itself.
Jon
> On Feb 20, 2018, at 1:10 PM, Carl Mueller <ca...@smartthings.com> wrote:
>
> When memtables/CommitLogs are flushed to disk/sstable, does the sstable go
> through sstable organization specific to each compaction strategy, or is
> the sstable creation the same for all compactionstrats and it is up to the
> compaction strategy to recompact the sstable if desired?
Re: Memtable flush -> SSTable: customizable or same for all
compaction strategies?
Posted by kurt greaves <ku...@instaclustr.com>.
>
> Also, I was wondering if the key cache maintains a count of how many local
> accesses a key undergoes. Such information might be very useful for
> compactions of sstables by splitting data by frequency of use so that those
> can be preferentially compacted.
No we don't currently have metrics for that, only overall cache
hits/misses. Measuring individual local accesses would probably have a
performance and memory impact but there's probably a way to do it
efficiently.
Has this been exploited... ever?
Not that I know of. I've theorised about using it previously with some
friends, but never got around to trying it. I imagine if you did you'd
probably have to fix some parts of the code to make it work (like
potentially discoverComponentsFor).
Typically I think any conversation that is relevant to the internals of
Cassandra is fine for the dev list, and that's the desired audience. Not
every dev watches the user list and only developers will really be able to
answer these questions. Lets face it, the dev list is pretty dead so not
sure why we care about a few emails landing there.
Re: Memtable flush -> SSTable: customizable or same for all
compaction strategies?
Posted by Carl Mueller <ca...@smartthings.com>.
Also, I was wondering if the key cache maintains a count of how many local
accesses a key undergoes. Such information might be very useful for
compactions of sstables by splitting data by frequency of use so that those
can be preferentially compacted.
On Wed, Feb 21, 2018 at 5:08 PM, Carl Mueller <ca...@smartthings.com>
wrote:
> Looking through the 2.1.X code I see this:
>
> org.apache.cassandra.io.sstable.Component.java
>
> In the enum for component types there is a CUSTOM enum value which seems
> to indicate a catchall for providing metadata for sstables.
>
> Has this been exploited... ever? I noticed in some of the patches for the
> archival options on TWCS there are complaints about being able to identify
> sstables that are archived and those that aren't.
>
> I would be interested in order to mark the sstables with metadata
> indicating the date range an sstable is targetted at for compactions.
>
> discoverComponentsFor seems to explicitly exclude the loadup of any
> files/sstable components that are CUSTOM in SStable.java
>
> On Wed, Feb 21, 2018 at 10:05 AM, Carl Mueller <
> carl.mueller@smartthings.com> wrote:
>
>> jon: I am planning on writing a custom compaction strategy. That's why
>> the question is here, I figured the specifics of memtable -> sstable and
>> cassandra internals are not a user question. If that still isn't deep
>> enough for the dev thread, I will move all those questions to user.
>>
>> On Wed, Feb 21, 2018 at 9:59 AM, Carl Mueller <
>> carl.mueller@smartthings.com> wrote:
>>
>>> Thank you all!
>>>
>>> On Tue, Feb 20, 2018 at 7:35 PM, kurt greaves <ku...@instaclustr.com>
>>> wrote:
>>>
>>>> Probably a lot of work but it would be incredibly useful for vnodes if
>>>> flushing was range aware (to be used with RangeAwareCompactionStrategy).
>>>> The writers are already range aware for JBOD, but that's not terribly
>>>> valuable ATM.
>>>>
>>>> On 20 February 2018 at 21:57, Jeff Jirsa <jj...@gmail.com> wrote:
>>>>
>>>>> There are some arguments to be made that the flush should consider
>>>>> compaction strategy - would allow a bug flush to respect LCS filesizes or
>>>>> break into smaller pieces to try to minimize range overlaps going from l0
>>>>> into l1, for example.
>>>>>
>>>>> I have no idea how much work would be involved, but may be worthwhile.
>>>>>
>>>>>
>>>>> --
>>>>> Jeff Jirsa
>>>>>
>>>>>
>>>>> On Feb 20, 2018, at 1:26 PM, Jon Haddad <jo...@jonhaddad.com> wrote:
>>>>>
>>>>> The file format is independent from compaction. A compaction strategy
>>>>> only selects sstables to be compacted, that’s it’s only job. It could have
>>>>> side effects, like generating other files, but any decent compaction
>>>>> strategy will account for the fact that those other files don’t exist.
>>>>>
>>>>> I wrote a blog post a few months ago going over some of the nuance of
>>>>> compaction you mind find informative: http://thelastpic
>>>>> kle.com/blog/2017/03/16/compaction-nuance.html
>>>>>
>>>>> This is also the wrong mailing list, please direct future user
>>>>> questions to the user list. The dev list is for development of Cassandra
>>>>> itself.
>>>>>
>>>>> Jon
>>>>>
>>>>> On Feb 20, 2018, at 1:10 PM, Carl Mueller <
>>>>> carl.mueller@smartthings.com> wrote:
>>>>>
>>>>> When memtables/CommitLogs are flushed to disk/sstable, does the
>>>>> sstable go
>>>>> through sstable organization specific to each compaction strategy, or
>>>>> is
>>>>> the sstable creation the same for all compactionstrats and it is up to
>>>>> the
>>>>> compaction strategy to recompact the sstable if desired?
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Memtable flush -> SSTable: customizable or same for all
compaction strategies?
Posted by Carl Mueller <ca...@smartthings.com>.
Looking through the 2.1.X code I see this:
org.apache.cassandra.io.sstable.Component.java
In the enum for component types there is a CUSTOM enum value which seems to
indicate a catchall for providing metadata for sstables.
Has this been exploited... ever? I noticed in some of the patches for the
archival options on TWCS there are complaints about being able to identify
sstables that are archived and those that aren't.
I would be interested in order to mark the sstables with metadata
indicating the date range an sstable is targetted at for compactions.
discoverComponentsFor seems to explicitly exclude the loadup of any
files/sstable components that are CUSTOM in SStable.java
On Wed, Feb 21, 2018 at 10:05 AM, Carl Mueller <carl.mueller@smartthings.com
> wrote:
> jon: I am planning on writing a custom compaction strategy. That's why the
> question is here, I figured the specifics of memtable -> sstable and
> cassandra internals are not a user question. If that still isn't deep
> enough for the dev thread, I will move all those questions to user.
>
> On Wed, Feb 21, 2018 at 9:59 AM, Carl Mueller <
> carl.mueller@smartthings.com> wrote:
>
>> Thank you all!
>>
>> On Tue, Feb 20, 2018 at 7:35 PM, kurt greaves <ku...@instaclustr.com>
>> wrote:
>>
>>> Probably a lot of work but it would be incredibly useful for vnodes if
>>> flushing was range aware (to be used with RangeAwareCompactionStrategy).
>>> The writers are already range aware for JBOD, but that's not terribly
>>> valuable ATM.
>>>
>>> On 20 February 2018 at 21:57, Jeff Jirsa <jj...@gmail.com> wrote:
>>>
>>>> There are some arguments to be made that the flush should consider
>>>> compaction strategy - would allow a bug flush to respect LCS filesizes or
>>>> break into smaller pieces to try to minimize range overlaps going from l0
>>>> into l1, for example.
>>>>
>>>> I have no idea how much work would be involved, but may be worthwhile.
>>>>
>>>>
>>>> --
>>>> Jeff Jirsa
>>>>
>>>>
>>>> On Feb 20, 2018, at 1:26 PM, Jon Haddad <jo...@jonhaddad.com> wrote:
>>>>
>>>> The file format is independent from compaction. A compaction strategy
>>>> only selects sstables to be compacted, that’s it’s only job. It could have
>>>> side effects, like generating other files, but any decent compaction
>>>> strategy will account for the fact that those other files don’t exist.
>>>>
>>>> I wrote a blog post a few months ago going over some of the nuance of
>>>> compaction you mind find informative: http://thelastpic
>>>> kle.com/blog/2017/03/16/compaction-nuance.html
>>>>
>>>> This is also the wrong mailing list, please direct future user
>>>> questions to the user list. The dev list is for development of Cassandra
>>>> itself.
>>>>
>>>> Jon
>>>>
>>>> On Feb 20, 2018, at 1:10 PM, Carl Mueller <ca...@smartthings.com>
>>>> wrote:
>>>>
>>>> When memtables/CommitLogs are flushed to disk/sstable, does the sstable
>>>> go
>>>> through sstable organization specific to each compaction strategy, or is
>>>> the sstable creation the same for all compactionstrats and it is up to
>>>> the
>>>> compaction strategy to recompact the sstable if desired?
>>>>
>>>>
>>>>
>>>
>>
>
Re: Memtable flush -> SSTable: customizable or same for all
compaction strategies?
Posted by Carl Mueller <ca...@smartthings.com>.
jon: I am planning on writing a custom compaction strategy. That's why the
question is here, I figured the specifics of memtable -> sstable and
cassandra internals are not a user question. If that still isn't deep
enough for the dev thread, I will move all those questions to user.
On Wed, Feb 21, 2018 at 9:59 AM, Carl Mueller <ca...@smartthings.com>
wrote:
> Thank you all!
>
> On Tue, Feb 20, 2018 at 7:35 PM, kurt greaves <ku...@instaclustr.com>
> wrote:
>
>> Probably a lot of work but it would be incredibly useful for vnodes if
>> flushing was range aware (to be used with RangeAwareCompactionStrategy).
>> The writers are already range aware for JBOD, but that's not terribly
>> valuable ATM.
>>
>> On 20 February 2018 at 21:57, Jeff Jirsa <jj...@gmail.com> wrote:
>>
>>> There are some arguments to be made that the flush should consider
>>> compaction strategy - would allow a bug flush to respect LCS filesizes or
>>> break into smaller pieces to try to minimize range overlaps going from l0
>>> into l1, for example.
>>>
>>> I have no idea how much work would be involved, but may be worthwhile.
>>>
>>>
>>> --
>>> Jeff Jirsa
>>>
>>>
>>> On Feb 20, 2018, at 1:26 PM, Jon Haddad <jo...@jonhaddad.com> wrote:
>>>
>>> The file format is independent from compaction. A compaction strategy
>>> only selects sstables to be compacted, that’s it’s only job. It could have
>>> side effects, like generating other files, but any decent compaction
>>> strategy will account for the fact that those other files don’t exist.
>>>
>>> I wrote a blog post a few months ago going over some of the nuance of
>>> compaction you mind find informative: http://thelastpic
>>> kle.com/blog/2017/03/16/compaction-nuance.html
>>>
>>> This is also the wrong mailing list, please direct future user questions
>>> to the user list. The dev list is for development of Cassandra itself.
>>>
>>> Jon
>>>
>>> On Feb 20, 2018, at 1:10 PM, Carl Mueller <ca...@smartthings.com>
>>> wrote:
>>>
>>> When memtables/CommitLogs are flushed to disk/sstable, does the sstable
>>> go
>>> through sstable organization specific to each compaction strategy, or is
>>> the sstable creation the same for all compactionstrats and it is up to
>>> the
>>> compaction strategy to recompact the sstable if desired?
>>>
>>>
>>>
>>
>
Re: Memtable flush -> SSTable: customizable or same for all
compaction strategies?
Posted by Carl Mueller <ca...@smartthings.com>.
Thank you all!
On Tue, Feb 20, 2018 at 7:35 PM, kurt greaves <ku...@instaclustr.com> wrote:
> Probably a lot of work but it would be incredibly useful for vnodes if
> flushing was range aware (to be used with RangeAwareCompactionStrategy).
> The writers are already range aware for JBOD, but that's not terribly
> valuable ATM.
>
> On 20 February 2018 at 21:57, Jeff Jirsa <jj...@gmail.com> wrote:
>
>> There are some arguments to be made that the flush should consider
>> compaction strategy - would allow a bug flush to respect LCS filesizes or
>> break into smaller pieces to try to minimize range overlaps going from l0
>> into l1, for example.
>>
>> I have no idea how much work would be involved, but may be worthwhile.
>>
>>
>> --
>> Jeff Jirsa
>>
>>
>> On Feb 20, 2018, at 1:26 PM, Jon Haddad <jo...@jonhaddad.com> wrote:
>>
>> The file format is independent from compaction. A compaction strategy
>> only selects sstables to be compacted, that’s it’s only job. It could have
>> side effects, like generating other files, but any decent compaction
>> strategy will account for the fact that those other files don’t exist.
>>
>> I wrote a blog post a few months ago going over some of the nuance of
>> compaction you mind find informative: http://thelastpic
>> kle.com/blog/2017/03/16/compaction-nuance.html
>>
>> This is also the wrong mailing list, please direct future user questions
>> to the user list. The dev list is for development of Cassandra itself.
>>
>> Jon
>>
>> On Feb 20, 2018, at 1:10 PM, Carl Mueller <ca...@smartthings.com>
>> wrote:
>>
>> When memtables/CommitLogs are flushed to disk/sstable, does the sstable go
>> through sstable organization specific to each compaction strategy, or is
>> the sstable creation the same for all compactionstrats and it is up to the
>> compaction strategy to recompact the sstable if desired?
>>
>>
>>
>
Re: Memtable flush -> SSTable: customizable or same for all
compaction strategies?
Posted by kurt greaves <ku...@instaclustr.com>.
Probably a lot of work but it would be incredibly useful for vnodes if
flushing was range aware (to be used with RangeAwareCompactionStrategy).
The writers are already range aware for JBOD, but that's not terribly
valuable ATM.
On 20 February 2018 at 21:57, Jeff Jirsa <jj...@gmail.com> wrote:
> There are some arguments to be made that the flush should consider
> compaction strategy - would allow a bug flush to respect LCS filesizes or
> break into smaller pieces to try to minimize range overlaps going from l0
> into l1, for example.
>
> I have no idea how much work would be involved, but may be worthwhile.
>
>
> --
> Jeff Jirsa
>
>
> On Feb 20, 2018, at 1:26 PM, Jon Haddad <jo...@jonhaddad.com> wrote:
>
> The file format is independent from compaction. A compaction strategy
> only selects sstables to be compacted, that’s it’s only job. It could have
> side effects, like generating other files, but any decent compaction
> strategy will account for the fact that those other files don’t exist.
>
> I wrote a blog post a few months ago going over some of the nuance of
> compaction you mind find informative: http://thelastpickle.com/blog/2017/
> 03/16/compaction-nuance.html
>
> This is also the wrong mailing list, please direct future user questions
> to the user list. The dev list is for development of Cassandra itself.
>
> Jon
>
> On Feb 20, 2018, at 1:10 PM, Carl Mueller <ca...@smartthings.com>
> wrote:
>
> When memtables/CommitLogs are flushed to disk/sstable, does the sstable go
> through sstable organization specific to each compaction strategy, or is
> the sstable creation the same for all compactionstrats and it is up to the
> compaction strategy to recompact the sstable if desired?
>
>
>
Re: Memtable flush -> SSTable: customizable or same for all compaction strategies?
Posted by Jeff Jirsa <jj...@gmail.com>.
There are some arguments to be made that the flush should consider compaction strategy - would allow a bug flush to respect LCS filesizes or break into smaller pieces to try to minimize range overlaps going from l0 into l1, for example.
I have no idea how much work would be involved, but may be worthwhile.
--
Jeff Jirsa
> On Feb 20, 2018, at 1:26 PM, Jon Haddad <jo...@jonhaddad.com> wrote:
>
> The file format is independent from compaction. A compaction strategy only selects sstables to be compacted, that’s it’s only job. It could have side effects, like generating other files, but any decent compaction strategy will account for the fact that those other files don’t exist.
>
> I wrote a blog post a few months ago going over some of the nuance of compaction you mind find informative: http://thelastpickle.com/blog/2017/03/16/compaction-nuance.html
>
> This is also the wrong mailing list, please direct future user questions to the user list. The dev list is for development of Cassandra itself.
>
> Jon
>
>> On Feb 20, 2018, at 1:10 PM, Carl Mueller <ca...@smartthings.com> wrote:
>>
>> When memtables/CommitLogs are flushed to disk/sstable, does the sstable go
>> through sstable organization specific to each compaction strategy, or is
>> the sstable creation the same for all compactionstrats and it is up to the
>> compaction strategy to recompact the sstable if desired?
>