You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Schubert Zhang (JIRA)" <ji...@apache.org> on 2010/05/01 10:19:53 UTC

[jira] Created: (CASSANDRA-1041) Skip large size (Configurable) SSTable in minor or/and major compaction

Skip large size (Configurable) SSTable in minor or/and major compaction
-----------------------------------------------------------------------

                 Key: CASSANDRA-1041
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1041
             Project: Cassandra
          Issue Type: New Feature
          Components: Core
            Reporter: Schubert Zhang
            Priority: Minor


When the SSTable files are large enough, such as 100GB, the compaction (include minor and major) cost is big (disk IO, CPU, memory), etc.
In some applications, we accept not compcating all SSTables to the final very large ones. 

This feature provide two optional configurable attributes MinorCompactSkipInGB and MajorCompactSkipInGB for each ColumnFamily. 
The optional MinorCompactSkipInGB attribute specifies the maximum size of SSTables which will be compcated in minor-compaction. The SSTables larger than MinorCompactSkipInGB will be skipped. The optional MajorCompactSkipInGB attribute is same for major-compaction.

The default of these attributes are 0, means do not skip, just as current 0.6.1.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-1041) Skip large size (Configurable) SSTable in minor or/and major compaction

Posted by "Schubert Zhang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Schubert Zhang updated CASSANDRA-1041:
--------------------------------------

    Attachment: CASSANDRA-1041-0.6.1.patch
                CASSANDRA-1041-0.6.patch

> Skip large size (Configurable) SSTable in minor or/and major compaction
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1041
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1041
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Schubert Zhang
>            Priority: Minor
>         Attachments: CASSANDRA-1041-0.6.1.patch, CASSANDRA-1041-0.6.patch
>
>
> When the SSTable files are large enough, such as 100GB, the compaction (include minor and major) cost is big (disk IO, CPU, memory), etc.
> In some applications, we accept not compcating all SSTables to the final very large ones. 
> This feature provide two optional configurable attributes MinorCompactSkipInGB and MajorCompactSkipInGB for each ColumnFamily. 
> The optional MinorCompactSkipInGB attribute specifies the maximum size of SSTables which will be compcated in minor-compaction. The SSTables larger than MinorCompactSkipInGB will be skipped. The optional MajorCompactSkipInGB attribute is same for major-compaction.
> The default of these attributes are 0, means do not skip, just as current 0.6.1.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-1041) Skip large size (Configurable) SSTable in minor or/and major compaction

Posted by "Schubert Zhang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Schubert Zhang updated CASSANDRA-1041:
--------------------------------------

    Attachment:     (was: CASSANDRA-1041-0.6.1.patch)

> Skip large size (Configurable) SSTable in minor or/and major compaction
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1041
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1041
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Schubert Zhang
>            Priority: Minor
>         Attachments: CASSANDRA-1041-0.6.1.patch
>
>
> When the SSTable files are large enough, such as 100GB, the compaction (include minor and major) cost is big (disk IO, CPU, memory), etc.
> In some applications, we accept not compcating all SSTables to the final very large ones. 
> This feature provide two optional configurable attributes MinorCompactSkipInGB and MajorCompactSkipInGB for each ColumnFamily. 
> The optional MinorCompactSkipInGB attribute specifies the maximum size of SSTables which will be compcated in minor-compaction. The SSTables larger than MinorCompactSkipInGB will be skipped. The optional MajorCompactSkipInGB attribute is same for major-compaction.
> The default of these attributes are 0, means do not skip, just as current 0.6.1.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1041) Skip large size (Configurable) SSTable in minor or/and major compaction

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12863553#action_12863553 ] 

Jonathan Ellis commented on CASSANDRA-1041:
-------------------------------------------

You don't want to skip large files in major compactions, since the definition of major is "compact everything so it is safe to remove tombstones."  (See http://wiki.apache.org/cassandra/DistributedDeletes and http://wiki.apache.org/cassandra/MemtableSSTable).

Major compactions are not run automatically so if you don't want that going on just don't invoke nodeprobe compact.

Minor compaction are batched up according to sstable size and only similar sizes are merged.  Here again ignoring larger sstables indefinitely is a bad idea, because the number of sstables you have to merge rows from for reads is unbounded.

Better solution: expose CompactionManager.minimumCompactionThreshold and CM.maximumCompactionThreshold in the configuration file, so you can tune minor compactions to require more or less sstables in the same "bucket" before compacting.


> Skip large size (Configurable) SSTable in minor or/and major compaction
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1041
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1041
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Schubert Zhang
>            Priority: Minor
>         Attachments: CASSANDRA-1041-0.6.1.patch, CASSANDRA-1041-0.6.patch
>
>
> When the SSTable files are large enough, such as 100GB, the compaction (include minor and major) cost is big (disk IO, CPU, memory), etc.
> In some applications, we accept not compcating all SSTables to the final very large ones. 
> This feature provide two optional configurable attributes MinorCompactSkipInGB and MajorCompactSkipInGB for each ColumnFamily. 
> The optional MinorCompactSkipInGB attribute specifies the maximum size of SSTables which will be compcated in minor-compaction. The SSTables larger than MinorCompactSkipInGB will be skipped. The optional MajorCompactSkipInGB attribute is same for major-compaction.
> The default of these attributes are 0, means do not skip, just as current 0.6.1.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-1041) Skip large size (Configurable) SSTable in minor or/and major compaction

Posted by "Schubert Zhang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Schubert Zhang updated CASSANDRA-1041:
--------------------------------------

    Attachment: CASSANDRA-1041-0.6.1.patch

Replace the CASSANDRA-1041-0.6.1.patch.

We found it is difficult distinguish major and minor compaction in current codebase.
So, only one optional attribute for ColumnFamily is provided here: CompactSkipInGB.

Maybe the use whose application need not delete operations can use this patch.

> Skip large size (Configurable) SSTable in minor or/and major compaction
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1041
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1041
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Schubert Zhang
>            Priority: Minor
>         Attachments: CASSANDRA-1041-0.6.1.patch
>
>
> When the SSTable files are large enough, such as 100GB, the compaction (include minor and major) cost is big (disk IO, CPU, memory), etc.
> In some applications, we accept not compcating all SSTables to the final very large ones. 
> This feature provide two optional configurable attributes MinorCompactSkipInGB and MajorCompactSkipInGB for each ColumnFamily. 
> The optional MinorCompactSkipInGB attribute specifies the maximum size of SSTables which will be compcated in minor-compaction. The SSTables larger than MinorCompactSkipInGB will be skipped. The optional MajorCompactSkipInGB attribute is same for major-compaction.
> The default of these attributes are 0, means do not skip, just as current 0.6.1.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-1041) Skip large size (Configurable) SSTable in minor or/and major compaction

Posted by "Schubert Zhang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Schubert Zhang updated CASSANDRA-1041:
--------------------------------------

    Attachment:     (was: CASSANDRA-1041-0.6.patch)

> Skip large size (Configurable) SSTable in minor or/and major compaction
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1041
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1041
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Schubert Zhang
>            Priority: Minor
>         Attachments: CASSANDRA-1041-0.6.1.patch
>
>
> When the SSTable files are large enough, such as 100GB, the compaction (include minor and major) cost is big (disk IO, CPU, memory), etc.
> In some applications, we accept not compcating all SSTables to the final very large ones. 
> This feature provide two optional configurable attributes MinorCompactSkipInGB and MajorCompactSkipInGB for each ColumnFamily. 
> The optional MinorCompactSkipInGB attribute specifies the maximum size of SSTables which will be compcated in minor-compaction. The SSTables larger than MinorCompactSkipInGB will be skipped. The optional MajorCompactSkipInGB attribute is same for major-compaction.
> The default of these attributes are 0, means do not skip, just as current 0.6.1.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1041) Skip large size (Configurable) SSTable in minor or/and major compaction

Posted by "Schubert Zhang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12863611#action_12863611 ] 

Schubert Zhang commented on CASSANDRA-1041:
-------------------------------------------

@Jonathan,
Yes, we understand that major compactions will clean the tombstones. We add this configuration based on following reasons:
(1) In same application, there is no delete operation or very seldom delete operation. 
(2) We still want to manually major compact old data to reduce the number of SSTables (but need not a single big one).
(3) In some filesystem, too big file gains inefficacy.

For minor compactions, "minimumCompactionThreshold"  and "maximumCompactionThreshold" are just the number of SSTables, but not size. We want use size threshold to avoid too many minor compactions in background, which cost too many disk IO and CPU and memory.

The tow configurable options are optional for users.

@Stu,
Yes, I also prefer to create multiple SSTables (rather than 1) according to size threshold (in our another project, it works well). But I think the "disjoint key ranges" is not so necessary in cassandra.

> Skip large size (Configurable) SSTable in minor or/and major compaction
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1041
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1041
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Schubert Zhang
>            Priority: Minor
>         Attachments: CASSANDRA-1041-0.6.1.patch, CASSANDRA-1041-0.6.patch
>
>
> When the SSTable files are large enough, such as 100GB, the compaction (include minor and major) cost is big (disk IO, CPU, memory), etc.
> In some applications, we accept not compcating all SSTables to the final very large ones. 
> This feature provide two optional configurable attributes MinorCompactSkipInGB and MajorCompactSkipInGB for each ColumnFamily. 
> The optional MinorCompactSkipInGB attribute specifies the maximum size of SSTables which will be compcated in minor-compaction. The SSTables larger than MinorCompactSkipInGB will be skipped. The optional MajorCompactSkipInGB attribute is same for major-compaction.
> The default of these attributes are 0, means do not skip, just as current 0.6.1.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-1041) Skip large size (Configurable) SSTable in minor or/and major compaction

Posted by "Schubert Zhang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Schubert Zhang updated CASSANDRA-1041:
--------------------------------------

    Attachment: CASSANDRA-1041-0.6.patch

The attached CASSANDRA-1041-0.6.patch is the patch base on 0.6 branch.

> Skip large size (Configurable) SSTable in minor or/and major compaction
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1041
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1041
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Schubert Zhang
>            Priority: Minor
>         Attachments: CASSANDRA-1041-0.6.1.patch, CASSANDRA-1041-0.6.patch
>
>
> When the SSTable files are large enough, such as 100GB, the compaction (include minor and major) cost is big (disk IO, CPU, memory), etc.
> In some applications, we accept not compcating all SSTables to the final very large ones. 
> This feature provide two optional configurable attributes MinorCompactSkipInGB and MajorCompactSkipInGB for each ColumnFamily. 
> The optional MinorCompactSkipInGB attribute specifies the maximum size of SSTables which will be compcated in minor-compaction. The SSTables larger than MinorCompactSkipInGB will be skipped. The optional MajorCompactSkipInGB attribute is same for major-compaction.
> The default of these attributes are 0, means do not skip, just as current 0.6.1.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-1041) Skip large size (Configurable) SSTable in minor or/and major compaction

Posted by "Schubert Zhang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Schubert Zhang updated CASSANDRA-1041:
--------------------------------------

    Attachment:     (was: CASSANDRA-1041-0.6.patch)

> Skip large size (Configurable) SSTable in minor or/and major compaction
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1041
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1041
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Schubert Zhang
>            Priority: Minor
>
> When the SSTable files are large enough, such as 100GB, the compaction (include minor and major) cost is big (disk IO, CPU, memory), etc.
> In some applications, we accept not compcating all SSTables to the final very large ones. 
> This feature provide two optional configurable attributes MinorCompactSkipInGB and MajorCompactSkipInGB for each ColumnFamily. 
> The optional MinorCompactSkipInGB attribute specifies the maximum size of SSTables which will be compcated in minor-compaction. The SSTables larger than MinorCompactSkipInGB will be skipped. The optional MajorCompactSkipInGB attribute is same for major-compaction.
> The default of these attributes are 0, means do not skip, just as current 0.6.1.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-1041) Skip large size (Configurable) SSTable in minor or/and major compaction

Posted by "Schubert Zhang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Schubert Zhang updated CASSANDRA-1041:
--------------------------------------

    Attachment:     (was: CASSANDRA-1041-0.6.1.patch)

> Skip large size (Configurable) SSTable in minor or/and major compaction
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1041
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1041
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Schubert Zhang
>            Priority: Minor
>
> When the SSTable files are large enough, such as 100GB, the compaction (include minor and major) cost is big (disk IO, CPU, memory), etc.
> In some applications, we accept not compcating all SSTables to the final very large ones. 
> This feature provide two optional configurable attributes MinorCompactSkipInGB and MajorCompactSkipInGB for each ColumnFamily. 
> The optional MinorCompactSkipInGB attribute specifies the maximum size of SSTables which will be compcated in minor-compaction. The SSTables larger than MinorCompactSkipInGB will be skipped. The optional MajorCompactSkipInGB attribute is same for major-compaction.
> The default of these attributes are 0, means do not skip, just as current 0.6.1.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-1041) Skip large size (Configurable) SSTable in minor or/and major compaction

Posted by "Schubert Zhang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Schubert Zhang updated CASSANDRA-1041:
--------------------------------------

    Attachment: CASSANDRA-1041-0.6.1.patch

My colleague White Liu and me modified the code of 0.6.1.

The attached "CASSANDRA-1041-0.6.1.patch" is a patch based on tag 0.6.1, and I have test it. 
If you want this small feature, please test and verify it. If it is necessary, we will patch it to 0.6.2 or later.

This is a example of configuration in storage-conf.xml

      <ColumnFamily Name="DATA"
                                    CompareWith="LongType"
                                    KeysCached="1000"
                                    MinorCompactSkipInGB="10"
                                    MajorCompactSkipInGB="100"/>

> Skip large size (Configurable) SSTable in minor or/and major compaction
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1041
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1041
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Schubert Zhang
>            Priority: Minor
>         Attachments: CASSANDRA-1041-0.6.1.patch
>
>
> When the SSTable files are large enough, such as 100GB, the compaction (include minor and major) cost is big (disk IO, CPU, memory), etc.
> In some applications, we accept not compcating all SSTables to the final very large ones. 
> This feature provide two optional configurable attributes MinorCompactSkipInGB and MajorCompactSkipInGB for each ColumnFamily. 
> The optional MinorCompactSkipInGB attribute specifies the maximum size of SSTables which will be compcated in minor-compaction. The SSTables larger than MinorCompactSkipInGB will be skipped. The optional MajorCompactSkipInGB attribute is same for major-compaction.
> The default of these attributes are 0, means do not skip, just as current 0.6.1.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1041) Skip large size (Configurable) SSTable in minor or/and major compaction

Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12863598#action_12863598 ] 

Stu Hood commented on CASSANDRA-1041:
-------------------------------------

An alternate solution has been raised before: partitioning the data stored on disk into multiple ranges.

For example, when a major compaction reaches a size threshold (as configured here), it creates N output SSTables (rather than 1) covering disjoint key ranges,  and records the range that each SSTable is responsible for. With N == 2 for instance, you'd have two sets of SSTables that could be major-compacted individually.

> Skip large size (Configurable) SSTable in minor or/and major compaction
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1041
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1041
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Schubert Zhang
>            Priority: Minor
>         Attachments: CASSANDRA-1041-0.6.1.patch, CASSANDRA-1041-0.6.patch
>
>
> When the SSTable files are large enough, such as 100GB, the compaction (include minor and major) cost is big (disk IO, CPU, memory), etc.
> In some applications, we accept not compcating all SSTables to the final very large ones. 
> This feature provide two optional configurable attributes MinorCompactSkipInGB and MajorCompactSkipInGB for each ColumnFamily. 
> The optional MinorCompactSkipInGB attribute specifies the maximum size of SSTables which will be compcated in minor-compaction. The SSTables larger than MinorCompactSkipInGB will be skipped. The optional MajorCompactSkipInGB attribute is same for major-compaction.
> The default of these attributes are 0, means do not skip, just as current 0.6.1.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1041) Skip large size (Configurable) SSTable in minor or/and major compaction

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12863626#action_12863626 ] 

Jonathan Ellis commented on CASSANDRA-1041:
-------------------------------------------

see CASSANDRA-16 for that

> Skip large size (Configurable) SSTable in minor or/and major compaction
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1041
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1041
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Schubert Zhang
>            Priority: Minor
>         Attachments: CASSANDRA-1041-0.6.1.patch, CASSANDRA-1041-0.6.patch
>
>
> When the SSTable files are large enough, such as 100GB, the compaction (include minor and major) cost is big (disk IO, CPU, memory), etc.
> In some applications, we accept not compcating all SSTables to the final very large ones. 
> This feature provide two optional configurable attributes MinorCompactSkipInGB and MajorCompactSkipInGB for each ColumnFamily. 
> The optional MinorCompactSkipInGB attribute specifies the maximum size of SSTables which will be compcated in minor-compaction. The SSTables larger than MinorCompactSkipInGB will be skipped. The optional MajorCompactSkipInGB attribute is same for major-compaction.
> The default of these attributes are 0, means do not skip, just as current 0.6.1.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-1041) Skip large size (Configurable) SSTable in minor or/and major compaction

Posted by "Schubert Zhang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Schubert Zhang updated CASSANDRA-1041:
--------------------------------------

    Attachment: CASSANDRA-1041-0.6.1.patch

> Skip large size (Configurable) SSTable in minor or/and major compaction
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1041
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1041
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Schubert Zhang
>            Priority: Minor
>         Attachments: CASSANDRA-1041-0.6.1.patch
>
>
> When the SSTable files are large enough, such as 100GB, the compaction (include minor and major) cost is big (disk IO, CPU, memory), etc.
> In some applications, we accept not compcating all SSTables to the final very large ones. 
> This feature provide two optional configurable attributes MinorCompactSkipInGB and MajorCompactSkipInGB for each ColumnFamily. 
> The optional MinorCompactSkipInGB attribute specifies the maximum size of SSTables which will be compcated in minor-compaction. The SSTables larger than MinorCompactSkipInGB will be skipped. The optional MajorCompactSkipInGB attribute is same for major-compaction.
> The default of these attributes are 0, means do not skip, just as current 0.6.1.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1041) Skip large size (Configurable) SSTable in minor or/and major compaction

Posted by "Schubert Zhang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12863641#action_12863641 ] 

Schubert Zhang commented on CASSANDRA-1041:
-------------------------------------------

@Jonathan,
I know and  understand all of what you pointed out all above. 
Could you please also give ous more advices for my "reasons" in my application. 
(1) Want to avoid cost of disk IO and CPU and memory usage when minor compaction.
(2) Do not want too big SSTable files, such as a file sized 1TB,  but several 100GB files are ok.
(3) Temporary work-round of the "whole row in memory" limitation. (I know the final fix of this limitation may be not ready immediately.) But it is really important.

> Skip large size (Configurable) SSTable in minor or/and major compaction
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1041
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1041
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Schubert Zhang
>            Priority: Minor
>         Attachments: CASSANDRA-1041-0.6.1.patch, CASSANDRA-1041-0.6.patch
>
>
> When the SSTable files are large enough, such as 100GB, the compaction (include minor and major) cost is big (disk IO, CPU, memory), etc.
> In some applications, we accept not compcating all SSTables to the final very large ones. 
> This feature provide two optional configurable attributes MinorCompactSkipInGB and MajorCompactSkipInGB for each ColumnFamily. 
> The optional MinorCompactSkipInGB attribute specifies the maximum size of SSTables which will be compcated in minor-compaction. The SSTables larger than MinorCompactSkipInGB will be skipped. The optional MajorCompactSkipInGB attribute is same for major-compaction.
> The default of these attributes are 0, means do not skip, just as current 0.6.1.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1041) Skip large size (Configurable) SSTable in minor or/and major compaction

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12863607#action_12863607 ] 

Jonathan Ellis commented on CASSANDRA-1041:
-------------------------------------------

As has also been raised before, that quickly gets messy as new data comes in that doesn't exactly distribute the way the first set did across the partition point.

> Skip large size (Configurable) SSTable in minor or/and major compaction
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1041
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1041
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Schubert Zhang
>            Priority: Minor
>         Attachments: CASSANDRA-1041-0.6.1.patch, CASSANDRA-1041-0.6.patch
>
>
> When the SSTable files are large enough, such as 100GB, the compaction (include minor and major) cost is big (disk IO, CPU, memory), etc.
> In some applications, we accept not compcating all SSTables to the final very large ones. 
> This feature provide two optional configurable attributes MinorCompactSkipInGB and MajorCompactSkipInGB for each ColumnFamily. 
> The optional MinorCompactSkipInGB attribute specifies the maximum size of SSTables which will be compcated in minor-compaction. The SSTables larger than MinorCompactSkipInGB will be skipped. The optional MajorCompactSkipInGB attribute is same for major-compaction.
> The default of these attributes are 0, means do not skip, just as current 0.6.1.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1041) Skip large size (Configurable) SSTable in minor or/and major compaction

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12863618#action_12863618 ] 

Jonathan Ellis commented on CASSANDRA-1041:
-------------------------------------------

the sstable parameters are the number _per bucket_, so the sstables you compact w/ minor alone decrease logarithmically as size increases

the right thing here is to make minor compactions work better automatically, not add extra configurables

> Skip large size (Configurable) SSTable in minor or/and major compaction
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1041
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1041
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Schubert Zhang
>            Priority: Minor
>         Attachments: CASSANDRA-1041-0.6.1.patch, CASSANDRA-1041-0.6.patch
>
>
> When the SSTable files are large enough, such as 100GB, the compaction (include minor and major) cost is big (disk IO, CPU, memory), etc.
> In some applications, we accept not compcating all SSTables to the final very large ones. 
> This feature provide two optional configurable attributes MinorCompactSkipInGB and MajorCompactSkipInGB for each ColumnFamily. 
> The optional MinorCompactSkipInGB attribute specifies the maximum size of SSTables which will be compcated in minor-compaction. The SSTables larger than MinorCompactSkipInGB will be skipped. The optional MajorCompactSkipInGB attribute is same for major-compaction.
> The default of these attributes are 0, means do not skip, just as current 0.6.1.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-1041) Skip large size (Configurable) SSTable in minor or/and major compaction

Posted by "Schubert Zhang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Schubert Zhang updated CASSANDRA-1041:
--------------------------------------

    Attachment:     (was: CASSANDRA-1041-0.6.1.patch)

> Skip large size (Configurable) SSTable in minor or/and major compaction
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1041
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1041
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Schubert Zhang
>            Priority: Minor
>
> When the SSTable files are large enough, such as 100GB, the compaction (include minor and major) cost is big (disk IO, CPU, memory), etc.
> In some applications, we accept not compcating all SSTables to the final very large ones. 
> This feature provide two optional configurable attributes MinorCompactSkipInGB and MajorCompactSkipInGB for each ColumnFamily. 
> The optional MinorCompactSkipInGB attribute specifies the maximum size of SSTables which will be compcated in minor-compaction. The SSTables larger than MinorCompactSkipInGB will be skipped. The optional MajorCompactSkipInGB attribute is same for major-compaction.
> The default of these attributes are 0, means do not skip, just as current 0.6.1.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1041) Skip large size (Configurable) SSTable in minor or/and major compaction

Posted by "Schubert Zhang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12863620#action_12863620 ] 

Schubert Zhang commented on CASSANDRA-1041:
-------------------------------------------

And another reason I add this size-threshold is because the limitation of current implementation: "Cassandra's compaction code currently deserializes an entire row (per columnfamily) at a time." 

Use a size-threshold, we can temporarily restrict the size of row not too large.

In fact, we are also trying to fix the limitation:  "Cassandra's compaction code currently deserializes an entire row (per columnfamily) at a time." 
Our imagine is iterate on columns in stead of rows. 

> Skip large size (Configurable) SSTable in minor or/and major compaction
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1041
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1041
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Schubert Zhang
>            Priority: Minor
>         Attachments: CASSANDRA-1041-0.6.1.patch, CASSANDRA-1041-0.6.patch
>
>
> When the SSTable files are large enough, such as 100GB, the compaction (include minor and major) cost is big (disk IO, CPU, memory), etc.
> In some applications, we accept not compcating all SSTables to the final very large ones. 
> This feature provide two optional configurable attributes MinorCompactSkipInGB and MajorCompactSkipInGB for each ColumnFamily. 
> The optional MinorCompactSkipInGB attribute specifies the maximum size of SSTables which will be compcated in minor-compaction. The SSTables larger than MinorCompactSkipInGB will be skipped. The optional MajorCompactSkipInGB attribute is same for major-compaction.
> The default of these attributes are 0, means do not skip, just as current 0.6.1.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.