You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2008/04/17 02:11:21 UTC

[jira] Created: (HBASE-586) HRegion runs HStore memcache snapshotting -- fix it so HRegion only knows about flushing

HRegion runs HStore memcache snapshotting -- fix it so HRegion only knows about flushing
----------------------------------------------------------------------------------------

                 Key: HBASE-586
                 URL: https://issues.apache.org/jira/browse/HBASE-586
             Project: Hadoop HBase
          Issue Type: Bug
            Reporter: stack
            Assignee: stack
            Priority: Critical
             Fix For: 0.2.0


HRegion needs to keep count of memcache sizes.  Currently its trying to run the  memcaches snapshots so it can keep up this number.  HRegion should know nothing about no memcache snapshotting -- should be internal to flushing ... especially since HBASE-532 fix.   HRegion can ask for store sizes....  Add that.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-586) HRegion runs HStore memcache snapshotting -- fix it so only HStore knows about workings of memcache

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591079#action_12591079 ] 

Jim Kellerman commented on HBASE-586:
-------------------------------------

Don't understand this part of javadoc for Memcache.getSnapshot:
{code}
   * Called by flusher when it wants to clean up snapshot made by a previous
   * call to {@link snapshot}
{code}

Otherwise +1


> HRegion runs HStore memcache snapshotting -- fix it so only HStore knows about workings of memcache
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-586
>                 URL: https://issues.apache.org/jira/browse/HBASE-586
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>            Priority: Critical
>             Fix For: 0.2.0
>
>         Attachments: 586-v2.patch, 586-v3.patch, 586.patch
>
>
> HRegion needs to keep count of memcache sizes.  Currently its trying to run the  memcaches snapshots so it can keep up this number.  HRegion should know nothing about no memcache snapshotting -- should be internal to flushing ... especially since HBASE-532 fix.   HRegion can ask for store sizes....  Add that.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-586) HRegion runs HStore memcache snapshotting -- fix it so only HStore knows about workings of memcache

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-586:
------------------------

       Resolution: Fixed
    Fix Version/s: 0.1.2
           Status: Resolved  (was: Patch Available)

Applied to TRUNK. Applied version without rewrite of TestMergeTool to branch after testing on cluster to make sure nothing broke.  Thanks for the review Jim.

> HRegion runs HStore memcache snapshotting -- fix it so only HStore knows about workings of memcache
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-586
>                 URL: https://issues.apache.org/jira/browse/HBASE-586
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>            Priority: Critical
>             Fix For: 0.2.0, 0.1.2
>
>         Attachments: 586-v2.patch, 586-v3.patch, 586.patch
>
>
> HRegion needs to keep count of memcache sizes.  Currently its trying to run the  memcaches snapshots so it can keep up this number.  HRegion should know nothing about no memcache snapshotting -- should be internal to flushing ... especially since HBASE-532 fix.   HRegion can ask for store sizes....  Add that.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-586) HRegion runs HStore memcache snapshotting -- fix it so only HStore knows about workings of memcache

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-586:
------------------------

    Status: Patch Available  (was: Open)

Submitting so I get some review action. 

Please pardon it's being a bit of an omnibus.

> HRegion runs HStore memcache snapshotting -- fix it so only HStore knows about workings of memcache
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-586
>                 URL: https://issues.apache.org/jira/browse/HBASE-586
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>            Priority: Critical
>             Fix For: 0.2.0
>
>         Attachments: 586-v2.patch, 586-v3.patch, 586.patch
>
>
> HRegion needs to keep count of memcache sizes.  Currently its trying to run the  memcaches snapshots so it can keep up this number.  HRegion should know nothing about no memcache snapshotting -- should be internal to flushing ... especially since HBASE-532 fix.   HRegion can ask for store sizes....  Add that.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-586) HRegion runs HStore memcache snapshotting -- fix it so HStore knows about workings of memcache

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-586:
------------------------

    Summary: HRegion runs HStore memcache snapshotting -- fix it so HStore knows about workings of memcache  (was: HRegion runs HStore memcache snapshotting -- fix it so HRegion only knows about flushing)

> HRegion runs HStore memcache snapshotting -- fix it so HStore knows about workings of memcache
> ----------------------------------------------------------------------------------------------
>
>                 Key: HBASE-586
>                 URL: https://issues.apache.org/jira/browse/HBASE-586
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>            Priority: Critical
>             Fix For: 0.2.0
>
>         Attachments: 586.patch
>
>
> HRegion needs to keep count of memcache sizes.  Currently its trying to run the  memcaches snapshots so it can keep up this number.  HRegion should know nothing about no memcache snapshotting -- should be internal to flushing ... especially since HBASE-532 fix.   HRegion can ask for store sizes....  Add that.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-586) HRegion runs HStore memcache snapshotting -- fix it so only HStore knows about workings of memcache

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-586:
------------------------

    Attachment: 586-v2.patch

Updated patch and updated commit message.  Need to run on cluster first to make sure new memcache sizing works under load.

{code}
HBASE-586 HRegion runs HStore memcache snapshotting -- fix it so only
HStore knows about workings of memcache

This patch moves the running of mecache snapshots out of the control
of HRegion and hides the memcache details in HStore.   This patch also
does a bunch of work on the merge tool fixing a bug in the metautils along
the way.  The merge tool test was failing.  We weren't setting into the
HLog the maximum sequence id after we'd opened a region -- as HRS does --
so were losing edits. On the way, refactored the merge tool test to get
rid of duplicated code.  Finally, cleans up logging in HStore to aid
debugging; e.g. we always refer to the sequence id as the 'sequence id'
in log messages rather than as seqId or 'sequence record' so can sort
log as see state of sequence id transitions.

Version 2 changes the order in which things are run in memcache.  532
made it so flushing did snapshot and then cleared the snapshot.  Now,
we snapshot before we flush a store, then inside in the store flush,
we call getSnapshot and then clearSnapshot.

M  src/test/org/apache/hadoop/hbase/regionserver/TestHMemcache.java
  How snapshotting changed.  Change test in accordance.
M  src/test/org/apache/hadoop/hbase/regionserver/TestHLog.java
  Use accessor to get file number.
M  src/test/org/apache/hadoop/hbase/util/TestMergeTool.java
  Refactored to remove duplicated code so could tell what was going on.
  (mergeAndVerify, verifyMerge): Addd.
M  src/java/org/apache/hadoop/hbase/regionserver/Memcache.java
  (snapshot): Changed so it no longer returns snapshot.
M  src/java/org/apache/hadoop/hbase/regionserver/HStore.java
  Changed log messages removing the useless and adding info to others.
  (this.maxSeqId): We used to add 1 to this in here in HStore.  Let
  HRegion do it.  Its the one that does the machinations w/ sequenceids
  anyways.  Make flushes return the amount flushed.  Use this updating
  the regions memcacheSize accounting.
  (snapshot): Added method for the region to call.
M  src/java/org/apache/hadoop/hbase/regionserver/HLog.java
  Made data members private.
  (getFileNum): Added accessor for tests.
M  src/java/org/apache/hadoop/hbase/regionserver/HRegion.java
  Added info to logs..
  (snapshotMemcaches): removed.
  (internalFlushcache): No longer takes startime.  Internally now does
  some of what used happen in snapshotMemcaches including sending of
  message to stores to snapshot.
  (getEntrySize): Added method for calculating size of an update. Used
  by HRegion and flushing so both come up w/ same answer.
M  src/java/org/apache/hadoop/hbase/util/Merge.java
  Add logging of whats happening during merges and fail earlier than we
  used if stuff is not right.
  Renamed local variables from region1 to r1, etc., so didn't clash
  with data members of same name.
M  src/java/org/apache/hadoop/hbase/util/MetaUtils.java
  Added a TODO
{code}

> HRegion runs HStore memcache snapshotting -- fix it so only HStore knows about workings of memcache
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-586
>                 URL: https://issues.apache.org/jira/browse/HBASE-586
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>            Priority: Critical
>             Fix For: 0.2.0
>
>         Attachments: 586-v2.patch, 586.patch
>
>
> HRegion needs to keep count of memcache sizes.  Currently its trying to run the  memcaches snapshots so it can keep up this number.  HRegion should know nothing about no memcache snapshotting -- should be internal to flushing ... especially since HBASE-532 fix.   HRegion can ask for store sizes....  Add that.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-586) HRegion runs HStore memcache snapshotting -- fix it so only HStore knows about workings of memcache

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591038#action_12591038 ] 

stack commented on HBASE-586:
-----------------------------

Oh, I ran this on cluster to prove that new way of doing region-level memstore size calculation is not going haywire.

> HRegion runs HStore memcache snapshotting -- fix it so only HStore knows about workings of memcache
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-586
>                 URL: https://issues.apache.org/jira/browse/HBASE-586
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>            Priority: Critical
>             Fix For: 0.2.0
>
>         Attachments: 586-v2.patch, 586-v3.patch, 586.patch
>
>
> HRegion needs to keep count of memcache sizes.  Currently its trying to run the  memcaches snapshots so it can keep up this number.  HRegion should know nothing about no memcache snapshotting -- should be internal to flushing ... especially since HBASE-532 fix.   HRegion can ask for store sizes....  Add that.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-586) HRegion runs HStore memcache snapshotting -- fix it so HRegion only knows about flushing

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-586:
------------------------

    Attachment: 586.patch

Here is first cut.  Merge is failing for some odd reason so need to fix that.  Also want to run this on cluster because its there where will figure if new technique actually works.

{code}
HBASE-586 HRegion runs HStore memcache snapshotting -- fix it so HRegion only knows about flushing
Make it so region does not know about store memcaches any more.  It
just knows about flushes.  Runs flushes, not flushes and memcache snapshots
individually.  Makes things a tad less complicated.

M  src/test/org/apache/hadoop/hbase/util/TestMergeTool.java
    Catch and report error if one running merge.
M  src/java/org/apache/hadoop/hbase/regionserver/HStore.java
    (snapshotMemcache): Removed.
    (flushCache): Added return of size flushed.
M  src/java/org/apache/hadoop/hbase/regionserver/HRegion.java
    (flushCache): Just call internalFlushcache.
    (snapshotMemcaches): Removed.
    (internalFlushcache): No longer take a 'starttime'.
    In here do things too like set the flushRequested flag and
    last flush time rather than over in the removed
    snapshotMemcaches.  Also, decrement memcacheSize by the
    amount of data flushed.
    (getEntrySize): Utility method.
M ndex: src/java/org/apache/hadoop/hbase/util/Merge.java
    Catch problems earlier.
{code}

> HRegion runs HStore memcache snapshotting -- fix it so HRegion only knows about flushing
> ----------------------------------------------------------------------------------------
>
>                 Key: HBASE-586
>                 URL: https://issues.apache.org/jira/browse/HBASE-586
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>            Priority: Critical
>             Fix For: 0.2.0
>
>         Attachments: 586.patch
>
>
> HRegion needs to keep count of memcache sizes.  Currently its trying to run the  memcaches snapshots so it can keep up this number.  HRegion should know nothing about no memcache snapshotting -- should be internal to flushing ... especially since HBASE-532 fix.   HRegion can ask for store sizes....  Add that.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-586) HRegion runs HStore memcache snapshotting -- fix it so only HStore knows about workings of memcache

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-586:
------------------------

    Attachment: 586-v3.patch

This version is like v2 but I did more clean up of logging messages adding method to FSUtils that will return a path minus the Filesystem instance qualifier.

> HRegion runs HStore memcache snapshotting -- fix it so only HStore knows about workings of memcache
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-586
>                 URL: https://issues.apache.org/jira/browse/HBASE-586
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>            Priority: Critical
>             Fix For: 0.2.0
>
>         Attachments: 586-v2.patch, 586-v3.patch, 586.patch
>
>
> HRegion needs to keep count of memcache sizes.  Currently its trying to run the  memcaches snapshots so it can keep up this number.  HRegion should know nothing about no memcache snapshotting -- should be internal to flushing ... especially since HBASE-532 fix.   HRegion can ask for store sizes....  Add that.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-586) HRegion runs HStore memcache snapshotting -- fix it so only HStore knows about workings of memcache

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-586:
------------------------

    Summary: HRegion runs HStore memcache snapshotting -- fix it so only HStore knows about workings of memcache  (was: HRegion runs HStore memcache snapshotting -- fix it so HStore knows about workings of memcache)

> HRegion runs HStore memcache snapshotting -- fix it so only HStore knows about workings of memcache
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-586
>                 URL: https://issues.apache.org/jira/browse/HBASE-586
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>            Priority: Critical
>             Fix For: 0.2.0
>
>         Attachments: 586.patch
>
>
> HRegion needs to keep count of memcache sizes.  Currently its trying to run the  memcaches snapshots so it can keep up this number.  HRegion should know nothing about no memcache snapshotting -- should be internal to flushing ... especially since HBASE-532 fix.   HRegion can ask for store sizes....  Add that.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.