You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2008/09/04 23:41:44 UTC

[jira] Created: (HBASE-871) Major configuration periodicity should be specifyable at the column family level, not cluster wide

Major configuration periodicity should be specifyable at the column family level, not cluster wide
--------------------------------------------------------------------------------------------------

                 Key: HBASE-871
                 URL: https://issues.apache.org/jira/browse/HBASE-871
             Project: Hadoop HBase
          Issue Type: Bug
            Reporter: stack


jon gray has a table of ten rows and a couple of columns that is constantly being updated.  Has max versions of 2.  This table is growing fast because all versions written are kept until a major compaction.  The way this table is being used is different than that of others.  Would be good if he could have major compactions run more often than the default once a day.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Assigned: (HBASE-871) Major configuration periodicity should be specifyable at the column family level, not cluster wide

Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Billy Pearson reassigned HBASE-871:
-----------------------------------

    Assignee: Billy Pearson  (was: Jonathan Gray)

> Major configuration periodicity should be specifyable at the column family level, not cluster wide
> --------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-871
>                 URL: https://issues.apache.org/jira/browse/HBASE-871
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: Billy Pearson
>         Attachments: hbase-871-v1.patch, hbase-871-v2.patch
>
>
> jon gray has a table of ten rows and a couple of columns that is constantly being updated.  Has max versions of 2.  This table is growing fast because all versions written are kept until a major compaction.  The way this table is being used is different than that of others.  Would be good if he could have major compactions run more often than the default once a day.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Assigned: (HBASE-871) Major configuration periodicity should be specifyable at the column family level, not cluster wide

Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Gray reassigned HBASE-871:
-----------------------------------

    Assignee: Jonathan Gray

> Major configuration periodicity should be specifyable at the column family level, not cluster wide
> --------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-871
>                 URL: https://issues.apache.org/jira/browse/HBASE-871
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: Jonathan Gray
>         Attachments: hbase-871-v1.patch
>
>
> jon gray has a table of ten rows and a couple of columns that is constantly being updated.  Has max versions of 2.  This table is growing fast because all versions written are kept until a major compaction.  The way this table is being used is different than that of others.  Would be good if he could have major compactions run more often than the default once a day.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-871) Major configuration periodicity should be specifyable at the column family level, not cluster wide

Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Gray updated HBASE-871:
--------------------------------

    Attachment: hbase-871-v2.patch

Previous patch did not compile.

HCD.getValue/setValue works only with Strings/byte[] so we must store period as String representation of long and do conversion when we fetch.

Could also add getLong/setLong to HCD but this will suffice for me.

> Major configuration periodicity should be specifyable at the column family level, not cluster wide
> --------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-871
>                 URL: https://issues.apache.org/jira/browse/HBASE-871
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: Jonathan Gray
>         Attachments: hbase-871-v1.patch, hbase-871-v2.patch
>
>
> jon gray has a table of ten rows and a couple of columns that is constantly being updated.  Has max versions of 2.  This table is growing fast because all versions written are kept until a major compaction.  The way this table is being used is different than that of others.  Would be good if he could have major compactions run more often than the default once a day.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-871) Major configuration periodicity should be specifyable at the column family level, not cluster wide

Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628690#action_12628690 ] 

Jonathan Gray commented on HBASE-871:
-------------------------------------

Confirmed that patch applies to 0.2 branch and trunk

> Major configuration periodicity should be specifyable at the column family level, not cluster wide
> --------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-871
>                 URL: https://issues.apache.org/jira/browse/HBASE-871
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: Jonathan Gray
>         Attachments: hbase-871-v1.patch
>
>
> jon gray has a table of ten rows and a couple of columns that is constantly being updated.  Has max versions of 2.  This table is growing fast because all versions written are kept until a major compaction.  The way this table is being used is different than that of others.  Would be good if he could have major compactions run more often than the default once a day.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Assigned: (HBASE-871) Major configuration periodicity should be specifyable at the column family level, not cluster wide

Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Billy Pearson reassigned HBASE-871:
-----------------------------------

    Assignee: Jonathan Gray  (was: Billy Pearson)

> Major configuration periodicity should be specifyable at the column family level, not cluster wide
> --------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-871
>                 URL: https://issues.apache.org/jira/browse/HBASE-871
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: Jonathan Gray
>         Attachments: hbase-871-v1.patch, hbase-871-v2.patch
>
>
> jon gray has a table of ten rows and a couple of columns that is constantly being updated.  Has max versions of 2.  This table is growing fast because all versions written are kept until a major compaction.  The way this table is being used is different than that of others.  Would be good if he could have major compactions run more often than the default once a day.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-871) Major configuration periodicity should be specifyable at the column family level, not cluster wide

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628490#action_12628490 ] 

stack commented on HBASE-871:
-----------------------------

A patch should do like the following. 

In the HStore constructor, it does this to get the major compaction time:

{code}
    this.majorCompactionTime = conf.getLong("hbase.hregion.majorcompaction", 86400000);
{code}

Just below it, it does this to get the config. for compression from HCD:

{code}
family.getCompression()
{code}

I'd think you'd do something like:

{code}
family.getValue("hbase.hregion.majorcompaction")
{code}

... if it came back null, you'd use the default.  Otherwise, use its value.

Add a define to HConstants for "hbase.hregion.majorcompaction" to make your life easier.

That takes care of the reading.

For setting it, the addtion to HConstants will help.  See the head of bin/HBase.rb.  See how we define constants there, constants that can then be used specifying stuff in the shell..

> Major configuration periodicity should be specifyable at the column family level, not cluster wide
> --------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-871
>                 URL: https://issues.apache.org/jira/browse/HBASE-871
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>
> jon gray has a table of ten rows and a couple of columns that is constantly being updated.  Has max versions of 2.  This table is growing fast because all versions written are kept until a major compaction.  The way this table is being used is different than that of others.  Would be good if he could have major compactions run more often than the default once a day.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-871) Major configuration periodicity should be specifyable at the column family level, not cluster wide

Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Gray updated HBASE-871:
--------------------------------

    Attachment: hbase-871-v1.patch

Per stacks recommendations, this patch just adds a check in HStore to see if a MAJOR_COMPACTION_PERIOD / hbase.hregion.majorcompaction was set for the family, and sets it if so.  Otherwise it remains at the default.

Also added MAJOR_COMPACTION_PERIOD to HConstants => "hbase.hregion.majorcompaction"

> Major configuration periodicity should be specifyable at the column family level, not cluster wide
> --------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-871
>                 URL: https://issues.apache.org/jira/browse/HBASE-871
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: Jonathan Gray
>         Attachments: hbase-871-v1.patch
>
>
> jon gray has a table of ten rows and a couple of columns that is constantly being updated.  Has max versions of 2.  This table is growing fast because all versions written are kept until a major compaction.  The way this table is being used is different than that of others.  Would be good if he could have major compactions run more often than the default once a day.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-871) Major compaction periodicity should be specifyable at the column family level, not cluster wide

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HBASE-871:
--------------------------------

    Summary: Major compaction periodicity should be specifyable at the column family level, not cluster wide  (was: Major configuration periodicity should be specifyable at the column family level, not cluster wide)

> Major compaction periodicity should be specifyable at the column family level, not cluster wide
> -----------------------------------------------------------------------------------------------
>
>                 Key: HBASE-871
>                 URL: https://issues.apache.org/jira/browse/HBASE-871
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: Jonathan Gray
>         Attachments: hbase-871-v1.patch, hbase-871-v2.patch
>
>
> jon gray has a table of ten rows and a couple of columns that is constantly being updated.  Has max versions of 2.  This table is growing fast because all versions written are kept until a major compaction.  The way this table is being used is different than that of others.  Would be good if he could have major compactions run more often than the default once a day.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-871) Major configuration periodicity should be specifyable at the column family level, not cluster wide

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628713#action_12628713 ] 

stack commented on HBASE-871:
-----------------------------

Suggest that this piece of your patch...

{code}
    if (family.getValue(MAJOR_COMPACTION_PERIOD) != null) {
      String strCompactionTime = family.getValue(HConstants.MAJOR_COMPACTION_PERIOD);
      this.majorCompactionTime = (new Long(strCompactionTime)).longValue();
    }
{code}

... go in under this line of code:

{code}
    this.majorCompactionTime = conf.getLong("hbase.hregion.majorcompaction", 86400000);
{code}

Also, use the new define in the line above now you've added it to HConstants.

We need to change the shell so you can pass arbitrary key/values to HCD and HTD but let that be another issue, HBASE-873

> Major configuration periodicity should be specifyable at the column family level, not cluster wide
> --------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-871
>                 URL: https://issues.apache.org/jira/browse/HBASE-871
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: Jonathan Gray
>         Attachments: hbase-871-v1.patch, hbase-871-v2.patch
>
>
> jon gray has a table of ten rows and a couple of columns that is constantly being updated.  Has max versions of 2.  This table is growing fast because all versions written are kept until a major compaction.  The way this table is being used is different than that of others.  Would be good if he could have major compactions run more often than the default once a day.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-871) Major compaction periodicity should be specifyable at the column family level, not cluster wide

Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Gray updated HBASE-871:
--------------------------------

    Attachment: hbase-871-v3.patch

Fixes described by stack

> Major compaction periodicity should be specifyable at the column family level, not cluster wide
> -----------------------------------------------------------------------------------------------
>
>                 Key: HBASE-871
>                 URL: https://issues.apache.org/jira/browse/HBASE-871
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.2.1, 0.18.0
>            Reporter: stack
>            Assignee: Jonathan Gray
>             Fix For: 0.2.1, 0.18.0
>
>         Attachments: hbase-871-v1.patch, hbase-871-v2.patch, hbase-871-v3.patch
>
>
> jon gray has a table of ten rows and a couple of columns that is constantly being updated.  Has max versions of 2.  This table is growing fast because all versions written are kept until a major compaction.  The way this table is being used is different than that of others.  Would be good if he could have major compactions run more often than the default once a day.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-871) Major compaction periodicity should be specifyable at the column family level, not cluster wide

Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Gray updated HBASE-871:
--------------------------------

        Fix Version/s: 0.18.0
                       0.2.1
    Affects Version/s: 0.18.0
                       0.2.1
               Status: Patch Available  (was: Open)

> Major compaction periodicity should be specifyable at the column family level, not cluster wide
> -----------------------------------------------------------------------------------------------
>
>                 Key: HBASE-871
>                 URL: https://issues.apache.org/jira/browse/HBASE-871
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.2.1, 0.18.0
>            Reporter: stack
>            Assignee: Jonathan Gray
>             Fix For: 0.2.1, 0.18.0
>
>         Attachments: hbase-871-v1.patch, hbase-871-v2.patch, hbase-871-v3.patch
>
>
> jon gray has a table of ten rows and a couple of columns that is constantly being updated.  Has max versions of 2.  This table is growing fast because all versions written are kept until a major compaction.  The way this table is being used is different than that of others.  Would be good if he could have major compactions run more often than the default once a day.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-871) Major compaction periodicity should be specifyable at the column family level, not cluster wide

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-871:
------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

Applied trunk and branch.  Thanks for patch Jon

> Major compaction periodicity should be specifyable at the column family level, not cluster wide
> -----------------------------------------------------------------------------------------------
>
>                 Key: HBASE-871
>                 URL: https://issues.apache.org/jira/browse/HBASE-871
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.2.1, 0.18.0
>            Reporter: stack
>            Assignee: Jonathan Gray
>             Fix For: 0.2.1, 0.18.0
>
>         Attachments: hbase-871-v1.patch, hbase-871-v2.patch, hbase-871-v3.patch
>
>
> jon gray has a table of ten rows and a couple of columns that is constantly being updated.  Has max versions of 2.  This table is growing fast because all versions written are kept until a major compaction.  The way this table is being used is different than that of others.  Would be good if he could have major compactions run more often than the default once a day.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.