You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "stack (JIRA)" <ji...@apache.org> on 2007/08/30 22:18:31 UTC

[jira] Created: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

[hbase] regionserver creates hlogs without bound
------------------------------------------------

                 Key: HADOOP-1820
                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
             Project: Hadoop
          Issue Type: Improvement
          Components: contrib/hbase
            Reporter: stack
            Priority: Minor


Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.

Just now, I had a host crash w/ 112 log files each of 30k plus edits each.

We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Attachment: patch.txt

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: Open  (was: Patch Available)

Forgot to cancel old patch first

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, HADOOP-1820-patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: Patch Available  (was: In Progress)

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, HADOOP-1820-patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530006 ] 

Hadoop QA commented on HADOOP-1820:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
http://issues.apache.org/jira/secure/attachment/12366483/patch.txt
against trunk revision r578964.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new compiler warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests -1.  The patch failed core unit tests.

    contrib tests +1.  The patch passed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/817/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/817/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/817/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/817/console

This message is automatically generated.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: Open  (was: Patch Available)

Each time it runs a different test fails that is unrelated to what this patch addresses

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HADOOP-1820:
--------------------------

    Attachment: excerpt.log

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: contrib/hbase
>            Reporter: stack
>            Priority: Minor
>         Attachments: excerpt.log
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: In Progress  (was: Patch Available)

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12531394 ] 

Hadoop QA commented on HADOOP-1820:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
http://issues.apache.org/jira/secure/attachment/12366825/patch.txt
against trunk revision r580745.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new compiler warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests +1.  The patch passed core unit tests.

    contrib tests +1.  The patch passed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/855/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/855/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/855/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/855/console

This message is automatically generated.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, HADOOP-1820-patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530240 ] 

Hudson commented on HADOOP-1820:
--------------------------------

Integrated in Hadoop-Nightly #250 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/250/])

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Attachment: patch.txt

Works locally. Try Hudson.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>            Reporter: stack
>            Assignee: Jim Kellerman
>         Attachments: excerpt.log, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12531585 ] 

Hudson commented on HADOOP-1820:
--------------------------------

Integrated in Hadoop-Nightly #256 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/256/])

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, HADOOP-1820-patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: Open  (was: Patch Available)

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Work started: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on HADOOP-1820 started by Jim Kellerman.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, HADOOP-1820-patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: Open  (was: Patch Available)

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: In Progress  (was: Patch Available)

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529789 ] 

Hadoop QA commented on HADOOP-1820:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
http://issues.apache.org/jira/secure/attachment/12366435/patch.txt
against trunk revision r578605.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new compiler warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests +1.  The patch passed core unit tests.

    contrib tests -1.  The patch failed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/813/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/813/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/813/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/813/console

This message is automatically generated.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529255 ] 

Hadoop QA commented on HADOOP-1820:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
http://issues.apache.org/jira/secure/attachment/12366305/patch.txt
against trunk revision r577603.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new compiler warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests +1.  The patch passed core unit tests.

    contrib tests -1.  The patch failed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/801/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/801/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/801/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/801/console

This message is automatically generated.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529442 ] 

Hadoop QA commented on HADOOP-1820:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
http://issues.apache.org/jira/secure/attachment/12366322/patch.txt
against trunk revision r577917.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new compiler warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests +1.  The patch passed core unit tests.

    contrib tests -1.  The patch failed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/804/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/804/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/804/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/804/console

This message is automatically generated.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529928 ] 

Hadoop QA commented on HADOOP-1820:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
http://issues.apache.org/jira/secure/attachment/12366435/patch.txt
against trunk revision r578879.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new compiler warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests +1.  The patch passed core unit tests.

    contrib tests -1.  The patch failed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/814/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/814/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/814/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/814/console

This message is automatically generated.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Issue Type: Bug  (was: Improvement)

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>            Reporter: stack
>            Assignee: Jim Kellerman
>         Attachments: excerpt.log
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: Patch Available  (was: Reopened)

Found and fixed possible deadlock condition.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, HADOOP-1820-patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: Patch Available  (was: Open)

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529765 ] 

Hadoop QA commented on HADOOP-1820:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
http://issues.apache.org/jira/secure/attachment/12366429/patch.txt
against trunk revision r578543.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new compiler warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests +1.  The patch passed core unit tests.

    contrib tests -1.  The patch failed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/811/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/811/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/811/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/811/console

This message is automatically generated.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Attachment: patch.txt

new patch based on recently committed changes for HADOOP-1923 and HADOOP-1924

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: Patch Available  (was: Open)

Works locally. Try Hudson.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: Patch Available  (was: Open)

Two different tests failed this time. Hudson is a Heisenbug generator.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Attachment: patch.txt

Allows the number of log files remaining at end to be <= 2 instead of exactly 2

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12527826 ] 

Hadoop QA commented on HADOOP-1820:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
http://issues.apache.org/jira/secure/attachment/12365927/patch.txt
against trunk revision r575986.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new compiler warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests +1.  The patch passed core unit tests.

    contrib tests -1.  The patch failed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/776/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/776/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/776/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/776/console

This message is automatically generated.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: Open  (was: Patch Available)

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529753 ] 

Jim Kellerman commented on HADOOP-1820:
---------------------------------------

There were three HBase tests that failed in build 804:

testLogRolling failed because a region split took an excessive amount of time (43 seconds), and by the time the split completed, the client had given up and started shutdown of the MiniHBase cluster. This is undoubtedly caused by the aggressive parameters we set for client retries (time between retries: 5 seconds, number of retries: 5) to make tests run faster. Because a region split is highly I/O intensive, the thread will often block on I/O. If the thread cannot get rescheduled because the client is doing a retry, this could cause the splitting thread to be starved and cause the long time for the split to complete. By contrast, the test testBasicSplit which does not have a cluster or a client running, completes a region split in 1 second.

Similarly a region split in testSplitRegionIsDeleted, which also runs a cluster and a client, took 50 seconds to complete a region split, and once again the client gave up and started shutting down the cluster just as the split was completing.

in testTableMapReduce, the default settings for region server timeout are too short so the master takes the region server out of rotation just before it completed its task.

So it appears that in the build environment, we need to increase the parameters for these respective tests so that they can run without timing out.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: Open  (was: Patch Available)

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12527834 ] 

Hadoop QA commented on HADOOP-1820:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
http://issues.apache.org/jira/secure/attachment/12365927/patch.txt
against trunk revision r575986.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new compiler warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests +1.  The patch passed core unit tests.

    contrib tests -1.  The patch failed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/777/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/777/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/777/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/777/console

This message is automatically generated.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Attachment: patch.txt

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

        Fix Version/s: 0.15.0
    Affects Version/s: 0.15.0
               Status: Patch Available  (was: Open)

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Attachment: patch.txt

Trap unexpected exceptions. Don't access dfs after it is shut down.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, HADOOP-1820-patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Reopened: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman reopened HADOOP-1820:
-----------------------------------


This test is failing on Hudson... reopening

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: Patch Available  (was: Open)

Works in my environment. Try Hudson

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, HADOOP-1820-patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: Patch Available  (was: Open)

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12527817 ] 

Hadoop QA commented on HADOOP-1820:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
http://issues.apache.org/jira/secure/attachment/12365927/patch.txt
against trunk revision r575986.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new compiler warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests +1.  The patch passed core unit tests.

    contrib tests -1.  The patch failed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/774/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/774/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/774/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/774/console

This message is automatically generated.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Attachment: patch.txt

Increase lease timeout period, reduce frequency of lease checking for TestLogRolling and TestSplit

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12527806 ] 

Hadoop QA commented on HADOOP-1820:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
http://issues.apache.org/jira/secure/attachment/12365926/patch.txt
against trunk revision r575986.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new compiler warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests +1.  The patch passed core unit tests.

    contrib tests -1.  The patch failed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/773/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/773/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/773/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/773/console

This message is automatically generated.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: Open  (was: Patch Available)

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, HADOOP-1820-patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: Patch Available  (was: In Progress)

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

Passed Hudson contrib tests (finally!). Committed.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: Patch Available  (was: Open)

Try with Hudson's upgraded JDK.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Attachment: patch.txt

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Attachment: patch.txt

Use default client ipc timeout in TestDFSAbort, wait longer in TestLogRolling for cache flushes and log rolls to complete.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

Tests passed. Committed.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, HADOOP-1820-patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: Patch Available  (was: In Progress)

This patch makes the client retry wait longer and the lease timeout longer for the effected tests.
Hopefully, it is slow enough that it will run on Hudson properly.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: Patch Available  (was: Open)

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12523938 ] 

stack commented on HADOOP-1820:
-------------------------------

Here's some more info.  The split process itself gets stuck never making it past file 2, throws an exception, and then starts over splitting from scratch again.  I'll attach a log file excerpt.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: contrib/hbase
>            Reporter: stack
>            Priority: Minor
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530896 ] 

Hadoop QA commented on HADOOP-1820:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
http://issues.apache.org/jira/secure/attachment/12366695/HADOOP-1820-patch.txt
against trunk revision r580166.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new compiler warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests +1.  The patch passed core unit tests.

    contrib tests -1.  The patch failed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/837/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/837/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/837/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/837/console

This message is automatically generated.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, HADOOP-1820-patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: In Progress  (was: Patch Available)

Almost from the very beginning TestHBaseCluster had problems with the mini dfs cluster:

    [junit] 07/09/23 21:09:13 WARN fs.FSNamesystem: Not able to place enough replicas, still in need of 1

This continued until the test timed out. Basically no forward progress could be made. The changes in this patch do not effect this test, so there must be some kind of race going on between the dfs mini cluster's servers and HBase's.

The next patch will back off some of the more aggressive test settings for this test to see if the real problem is timing related. Since the tests that failed previously succeeded this time, apparently, for some tests, the aggressive parameters used for testing are just a bit too aggressive for a machine which is "resource challenged" as hudson appears to be.


> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530428 ] 

Hudson commented on HADOOP-1820:
--------------------------------

Integrated in Hadoop-Nightly #251 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/251/])

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529770 ] 

Hadoop QA commented on HADOOP-1820:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
http://issues.apache.org/jira/secure/attachment/12366432/patch.txt
against trunk revision r578605.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new compiler warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests +1.  The patch passed core unit tests.

    contrib tests -1.  The patch failed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/812/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/812/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/812/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/812/console

This message is automatically generated.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Assigned: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman reassigned HADOOP-1820:
-------------------------------------

    Assignee: Jim Kellerman

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: contrib/hbase
>            Reporter: stack
>            Assignee: Jim Kellerman
>            Priority: Minor
>         Attachments: excerpt.log
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: Patch Available  (was: In Progress)

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Attachment: HADOOP-1820-patch.txt

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, HADOOP-1820-patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: Open  (was: Patch Available)

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: Patch Available  (was: In Progress)

TestLogRolling: wait after writing until data is flushed to disk.

TestHBaseCluster: Make lease timeout longer, increase time between client retries.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Priority: Major  (was: Minor)

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: contrib/hbase
>            Reporter: stack
>            Assignee: Jim Kellerman
>         Attachments: excerpt.log
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12526303 ] 

Jim Kellerman commented on HADOOP-1820:
---------------------------------------

Every time the log is rolled, it is supposed to delete logs which are older than the oldest applied edit.

It could be that when a region splits, the information about the parent region is not getting removed from the HLog's map of region name -> last commit id. Since the parent region will never receive another update, its sequence number will continue to become older and older and prevent any logs from being deleted since the last sequence number for the oldest parent region. Consequently the only way the logs will get cleaned up is if:

- the region server shuts down cleanly
- the region server crashes and the master splits the logs for it.

it appears that what we need to do is when a region splits, we need to remove it from the map of region names -> last sequence number.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: contrib/hbase
>            Reporter: stack
>            Assignee: Jim Kellerman
>            Priority: Minor
>         Attachments: excerpt.log
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1820) [hbase] regionserver creates hlogs without bound

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-1820:
----------------------------------

    Status: In Progress  (was: Patch Available)

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period, the log is rotated and a new one is opened.  While the region server is up, the logs grow in number without bound.  Only the most recent contain unpersisted edits.  If the region server goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery, the logs of edits are sorted and split  per region.  Recovery would run faster if it did not have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs, it could check for edit logs to clean.  When rolled, logs could be marked with the sequence id of their last contained edit.  The thread could on a period ask each hosted region for the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.