You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Lars Hofhansl (JIRA)" <ji...@apache.org> on 2012/09/15 06:23:10 UTC

[jira] [Created] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Lars Hofhansl created HBASE-6796:
------------------------------------

             Summary: Backport HBASE-5547, Don't delete HFiles in backup mode.
                 Key: HBASE-6796
                 URL: https://issues.apache.org/jira/browse/HBASE-6796
             Project: HBase
          Issue Type: Bug
            Reporter: Lars Hofhansl
            Assignee: Jesse Yates
             Fix For: 0.94.3




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Jesse Yates (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jesse Yates updated HBASE-6796:
-------------------------------

    Attachment: hbase-6796-v2.patch

Attaching patch on top of current 0.94 branch. Ran "mvn clean test -Dtest=TestLogsCleaner,TestHFileArchiving,TesstCatalogJanitor,TestCleanerChore,TestHFileCleaner,TestFSTableDescriptors,TestHFileArchiveUtil -P localTests" with no test failures.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489094#comment-13489094 ] 

Hudson commented on HBASE-6796:
-------------------------------

Integrated in HBase-0.94 #565 (See [https://builds.apache.org/job/HBase-0.94/565/])
    HBASE-6796 Backport HBASE-5547, Don't delete HFiles in backup mode. (Jesse Yates) (Revision 1404762)

     Result = FAILURE
larsh : 
Files : 
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/BaseConfigurable.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/Chore.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/backup
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/LogCleaner.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/LogCleanerDelegate.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/TimeToLiveLogCleaner.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveHFileCleaner.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveLogCleaner.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
* /hbase/branches/0.94/src/main/resources/hbase-default.xml
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/backup
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestLogsCleaner.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/cleaner
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestCleanerChore.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestLogsCleaner.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/CheckedArchivingHFileCleaner.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHFileArchiveUtil.java

                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Jesse Yates (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485695#comment-13485695 ] 

Jesse Yates commented on HBASE-6796:
------------------------------------

[~lhofhansl] what do you think, wanna pull this into 0.94.3?
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ted Yu reassigned HBASE-6796:
-----------------------------

    Assignee: Ted Yu  (was: Jesse Yates)
    
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Ted Yu
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456548#comment-13456548 ] 

stack commented on HBASE-6796:
------------------------------

[~ted_yu] Regards "I think we should first make sure related unit tests in trunk pass consistently before backporting."  Please name them.  Also, stop with the shape-shifting obstruction (What a minor feature is, when 0.96 will RC).  I can see why Lars would want to backport.  I do not know what your motive for obstructing the backport is other than a quote from the Book of Todd (slightly relevant but from another, more general context).  Quit too w/ the telling folks what to do: "....We should put more energy on 0.96."  Its a volunteer project.  And you are saying this to the 0.94 RM.  -1 with a reason or get out of the way of fellas trying to do a bit of work.

0.96, I'd like to branch soon.  There ain't that many blockers left and some I can punt I'd say -- they don't belong.  I'd like to run it for a while to make sure it basically works... then branch.  Two weeks?  Four weeks?  Then harden the branch to see if could cut an RC... another two to four weeks presuming I don't find anything crazy?  RC in two months?  SWAG!

0.96 is the singularity too.

I took a quick look at the patch.  Its big adding a bunch of stuff but a bunch of the bulk comes from moving files it seems.  Is the archiving off by default?  You fellas had much experience running this patch?




                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456318#comment-13456318 ] 

Ted Yu commented on HBASE-6796:
-------------------------------

May I ask what would be the indication for judging a feature being major or minor ?

I went though Jesse's backport one more time. The patch touches Master, HRegion, Store. Some classes were removed, some abstract classes added.
I would say this is not a minor feature.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Jesse Yates (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490461#comment-13490461 ] 

Jesse Yates commented on HBASE-6796:
------------------------------------

Sorry I've been AWOL on this - caught up in work/life. I think Lars' change should handle the problem, though file manipulation/creation _should_ be a fast operation - guess not up on Jenkins :) Looks like the test has been passing since though. thanks for the follow up Lars.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lars Hofhansl resolved HBASE-6796.
----------------------------------

      Resolution: Fixed
    Hadoop Flags: Reviewed

Committed to 0.94... Pfeeww.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467462#comment-13467462 ] 

Lars Hofhansl commented on HBASE-6796:
--------------------------------------

Yes, Ted, we will fix them before this gets committed.
These are all test fixes as far as I can tell.

                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489012#comment-13489012 ] 

Lars Hofhansl commented on HBASE-6796:
--------------------------------------

Or better even, use a better testdir for that test.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488468#comment-13488468 ] 

Lars Hofhansl commented on HBASE-6796:
--------------------------------------

I am going to commit tomorrow unless I hear objections.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456693#comment-13456693 ] 

stack commented on HBASE-6796:
------------------------------

I'd be fine w/ your committing this after running it a while internally just to make sure it basically works.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Jesse Yates (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488933#comment-13488933 ] 

Jesse Yates commented on HBASE-6796:
------------------------------------

[~lhofhansl] seemed like it would be less work for you this way.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Jesse Yates (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489010#comment-13489010 ] 

Jesse Yates commented on HBASE-6796:
------------------------------------

Hmm, I added "testReadingArchiveDirectoryFromFS()". And that method literally has nothing to do with the added code. Looks like a flapper already in the codebase. 

On a related note, it looks like Jenkins isn't running all the tests for 0.94. See https://builds.apache.org/job/HBase-0.94/563/testReport/org.apache.hadoop.hbase.util/ which should have run TestFSTableDescriptors...but shows neither hide nor hair of it (or many other expected tests).
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488929#comment-13488929 ] 

Lars Hofhansl commented on HBASE-6796:
--------------------------------------

Wanna do it this way, or should I rather commit HBASE-7082 to 0.94? I'm fine either way.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489921#comment-13489921 ] 

Ted Yu commented on HBASE-6796:
-------------------------------

Good finding, Lars.
I guess it was a typo (missing a 0).

>From https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.94/567/testReport/org.apache.hadoop.hbase.master.cleaner/TestHFileCleaner/testHFileCleaning/:
{code}
2012-11-02 20:56:40,589 INFO  [pool-1-thread-1] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testHFileCleaning: 45 threads (was 44), 127 file descriptors (was 127). 0 connections,  -thread leak?- 
2012-11-02 20:56:40,590 INFO  [pool-1-thread-1] hbase.ResourceChecker(180): after master.cleaner.TestHFileCleaner#testHFileCleaning: potentially hanging thread
2012-11-02 20:56:40,591 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	java.lang.Object.wait(Native Method)
2012-11-02 20:56:40,591 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	java.lang.Object.wait(Object.java:485)
2012-11-02 20:56:40,591 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	org.apache.hadoop.ipc.Client.call(Client.java:1056)
2012-11-02 20:56:40,591 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
2012-11-02 20:56:40,591 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	$Proxy9.create(Unknown Source)
2012-11-02 20:56:40,591 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
2012-11-02 20:56:40,591 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
2012-11-02 20:56:40,591 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	java.lang.reflect.Method.invoke(Method.java:597)
2012-11-02 20:56:40,592 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
2012-11-02 20:56:40,592 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
2012-11-02 20:56:40,592 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	$Proxy9.create(Unknown Source)
2012-11-02 20:56:40,592 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:3248)
2012-11-02 20:56:40,592 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:713)
2012-11-02 20:56:40,592 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:182)
2012-11-02 20:56:40,592 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
2012-11-02 20:56:40,592 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
2012-11-02 20:56:40,593 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	org.apache.hadoop.fs.FileSystem.create(FileSystem.java:498)
2012-11-02 20:56:40,593 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	org.apache.hadoop.fs.FileSystem.createNewFile(FileSystem.java:638)
2012-11-02 20:56:40,593 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	org.apache.hadoop.hbase.master.cleaner.TestHFileCleaner.testHFileCleaning(TestHFileCleaner.java:119)
2012-11-02 20:56:40,593 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2012-11-02 20:56:40,593 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
2012-11-02 20:56:40,593 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
2012-11-02 20:56:40,593 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	java.lang.reflect.Method.invoke(Method.java:597)
2012-11-02 20:56:40,593 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
2012-11-02 20:56:40,594 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
2012-11-02 20:56:40,594 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
2012-11-02 20:56:40,594 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
2012-11-02 20:56:40,594 INFO  [pool-1-thread-1] hbase.ResourceChecker(186): 	org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)
{code}
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ted Yu reassigned HBASE-6796:
-----------------------------

    Assignee: Jesse Yates  (was: Ted Yu)

Resource checker would report thread(s) which aren't terminated after the timeout. 
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456312#comment-13456312 ] 

Ted Yu commented on HBASE-6796:
-------------------------------

bq. would consider this an safety feature.
I think you meant safe feature.

I think we should first make sure related unit tests in trunk pass consistently before backporting.

Allow me to quote Todd:
bq. I'm more OK with putting minor features in 0.94.x

I doubt a 147KB patch represents a minor feature.

bq. I think the Apache project should be concerned with pushing the ball forward on new releases and not on backporting features to older branches.
I like the above quote. Looks like 0.96.0 RC0 isn't that far from us. We should put more energy on 0.96

Just me two cents.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456320#comment-13456320 ] 

Ted Yu commented on HBASE-6796:
-------------------------------

bq. and no 0.96 release is in sight
I tend to disagree with the above assessment.
Now that Gary has nailed HBASE-5448, I would say the shape of 0.96 is getting clearer in terms of features.
We just need to tackle the outstanding blockers and critical bugs.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489919#comment-13489919 ] 

Lars Hofhansl commented on HBASE-6796:
--------------------------------------

In trunk there no 2000ms annotation on that test.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lars Hofhansl updated HBASE-6796:
---------------------------------

    Attachment: hbase-5547-0.94-backport-v0.patch

Jesse's patch.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467438#comment-13467438 ] 

stack commented on HBASE-6796:
------------------------------

+1 on commit then
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456560#comment-13456560 ] 

Ted Yu commented on HBASE-6796:
-------------------------------

@Stack:
There are currently 3 open Jiras which are linked to this issue. 
I exchanged several emails with Lars and he agrees to tackle these 3 issues before back porting. 
Since this feature is useful to users, I am in support of back poring. 

Regards
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489105#comment-13489105 ] 

Lars Hofhansl commented on HBASE-6796:
--------------------------------------

That test run didn't look so good.
[~jesse_yates] Mind having a look?
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489003#comment-13489003 ] 

Lars Hofhansl commented on HBASE-6796:
--------------------------------------

And of course it only happens sometimes and never in the debugger. Sigh.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Jesse Yates (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jesse Yates updated HBASE-6796:
-------------------------------

    Attachment: hbase-6796-v0.patch

Attaching patch for backport. Its doesn't include any of the backup.examples classes (e.g. ZooKeeperTableArchiveClient and friends) since that is really not all the important and has been a notoriously flaky test.

Includes:
* Port of HBASE-6797 - TestHFileCleaner fails
** set file mod times rather than sleeping when testing TTL
* Port of HBASE-6439- ignore .archive as TL directory
* Slight cleanup of testhfilecleaner
** Makes sure the file to save falls wthin the TTL
* Applying HBASE-6949: Automatically remove directories via CleanerChore.
** Had to add in local StoppableImplementation.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489932#comment-13489932 ] 

Hudson commented on HBASE-6796:
-------------------------------

Integrated in HBase-0.94 #568 (See [https://builds.apache.org/job/HBase-0.94/568/])
    HBASE-6796 ADDENDUM, remove spurious time limit from testHFileCleaning (Revision 1405275)

     Result = FAILURE
larsh : 
Files : 
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java

                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456491#comment-13456491 ] 

Lars Hofhansl commented on HBASE-6796:
--------------------------------------

@Stack: We will have needed it at Salesforce if an admin type person accidentally dropped a table... Unless that happens we will not have needed it. :)

Seriously though, I think for a database these data retention policies are essential. So the question is: Is the added data safety outweighed by the risk introduced by new code?

Maybe Salesforce is special here. Our capital is that we never lose data. Period.
If our needs do not align with the community here, we could obviously maintain this as private patch on top of 0.94. But for obvious reasons I am trying to avoid that.

@Stack: What's your thinking about an 0.96 timeline? When do you think it would be stable?
(I know I am asking for a shot in the dark here. There are 173 open issues against it still, so it's hard to say?)

@Ted, I absolutely agree with you, that we need to track down the test failures in trunk before this can considered for backporting.

@Jesse, can you have a look at HBASE-6439, HBASE-6797, and HBASE-6707?

                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489019#comment-13489019 ] 

Lars Hofhansl commented on HBASE-6796:
--------------------------------------

According to HBASE-4737 Medium should be < 45s. So I'll make it a LargeTest and then commit... Finally.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Comment Edited] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489221#comment-13489221 ] 

Ted Yu edited comment on HBASE-6796 at 11/2/12 3:37 AM:
--------------------------------------------------------

I tried to reproduce the test failure in TestHFileCleaner#testHFileCleaning but was not successful.
I looked at the test output:
https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.94/566/testReport/org.apache.hadoop.hbase.master.cleaner/TestHFileCleaner/testHFileCleaning/
{code}
2012-11-02 01:29:55,534 INFO  [pool-1-thread-1] hbase.ResourceChecker(145): before master.cleaner.TestHFileCleaner#testTTLCleaner: 43 threads, 127 file descriptors 0 connections, 
2012-11-02 01:29:55,898 DEBUG [pool-1-thread-1] cleaner.TimeToLiveHFileCleaner(68): Life:108, ttl:100, current:1351819795897, from: 1351819795789
2012-11-02 01:29:55,900 INFO  [pool-1-thread-1] hbase.ResourceChecker(145): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 127 file descriptors (was 127). 0 connections,  -thread leak?- 
{code}
We can see that one thread didn't exit.

Still, it is not very obvious which thread hung :-(
                
      was (Author: yuzhihong@gmail.com):
    I tried to reproduce the test failure in TestHFileCleaner#testHFileCleaning but was not successful.
I looked at the test output:
https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.94/566/testReport/org.apache.hadoop.hbase.master.cleaner/TestHFileCleaner/testHFileCleaning/
{code}
2012-11-02 01:29:57,917 INFO  [pool-1-thread-1] hbase.ResourceChecker(145): before master.cleaner.TestHFileCleaner#testRemovesEmptyDirectories: 45 threads, 127 file descriptors 0 connections, 
...
2012-11-02 01:29:58,730 INFO  [pool-1-thread-1] hbase.ResourceChecker(145): after master.cleaner.TestHFileCleaner#testRemovesEmptyDirectories: 44 threads (was 45), 127 file descriptors (was 127). 0 connections, 
{code}
We can see that one thread didn't exit.
I checked test output on MacBook. The thread count in both occasions was the same.

Still, it is not very obvious which thread hung :-(
                  
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489011#comment-13489011 ] 

Lars Hofhansl commented on HBASE-6796:
--------------------------------------

Test works if the datadir is cleaned first (either in an @After method or inline in this one test.)
I'll just paste that fix on this patch and commit.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Jesse Yates (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456636#comment-13456636 ] 

Jesse Yates commented on HBASE-6796:
------------------------------------

bq. Address the three issues mentioned above (HBASE-6439, HBASE-6797, HBASE-6707)

* There is a patch for 6707 already, just need a committer to go take a look.
* HBASE-6439 is really more of a nice to have - it just prints an warning message occasionally and not needed for correctness. I don't feel its necessary before a backport, but if people are vehement on this, I'll get behind it too

bq. Run with this patch on our test cluster and poke some load at it, delete some large tables, etc. (so that it leads to lots of deleted HFiles)

Definitely +1 on this. Probably end up with some good backup verification tools from this too.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489017#comment-13489017 ] 

Lars Hofhansl commented on HBASE-6796:
--------------------------------------

One last question: Should TestHFileArchiving be a LargeTest? It runs for 122s on my machine.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489221#comment-13489221 ] 

Ted Yu commented on HBASE-6796:
-------------------------------

I tried to reproduce the test failure in TestHFileCleaner#testHFileCleaning but was not successful.
I looked at the test output:
https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.94/566/testReport/org.apache.hadoop.hbase.master.cleaner/TestHFileCleaner/testHFileCleaning/
{code}
2012-11-02 01:29:57,917 INFO  [pool-1-thread-1] hbase.ResourceChecker(145): before master.cleaner.TestHFileCleaner#testRemovesEmptyDirectories: 45 threads, 127 file descriptors 0 connections, 
...
2012-11-02 01:29:58,730 INFO  [pool-1-thread-1] hbase.ResourceChecker(145): after master.cleaner.TestHFileCleaner#testRemovesEmptyDirectories: 44 threads (was 45), 127 file descriptors (was 127). 0 connections, 
{code}
We can see that one thread didn't exit.
I checked test output on MacBook. The thread count in both occasions was the same.

Still, it is not very obvious which thread hung :-(
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467378#comment-13467378 ] 

Lars Hofhansl commented on HBASE-6796:
--------------------------------------

I ran this on our cluster, created a table with 1bn rows, dropped the table, and checked that the HFile archiving does not fall over (either during the creation or dropping the table afterwards).

                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456315#comment-13456315 ] 

Lars Hofhansl commented on HBASE-6796:
--------------------------------------

Yeah, but 0.94 is the current release, and no 0.96 release is in sight.

The size of the patch is not necessarily an indication whether the feature is major or minor.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488936#comment-13488936 ] 

Lars Hofhansl commented on HBASE-6796:
--------------------------------------

Hmm... Patch does not apply cleanly to 0.94
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Jesse Yates (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490463#comment-13490463 ] 

Jesse Yates commented on HBASE-6796:
------------------------------------

and Ted :)
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489561#comment-13489561 ] 

Ted Yu commented on HBASE-6796:
-------------------------------

It turns out that there is one thread hanging, even on MacBook, after TestHFileCleaner#testTTLCleaner finishes.
With the help of patch from HBASE-7086, I saw:
{code}
2012-11-02 10:14:05,485 INFO  [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections,  -thread leak?-

2012-11-02 10:14:05,485 INFO  [main] hbase.ResourceChecker(179): after master.cleaner.TestHFileCleaner#testTTLCleaner: potentially hanging thread

2012-11-02 10:14:05,486 INFO  [main] hbase.ResourceChecker(185):  java.lang.Thread.sleep(Native Method)

2012-11-02 10:14:05,487 INFO  [main] hbase.ResourceChecker(185):  org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1379)

2012-11-02 10:14:05,487 INFO  [main] hbase.ResourceChecker(185):  java.lang.Thread.run(Thread.java:680)
{code}
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456321#comment-13456321 ] 

Lars Hofhansl commented on HBASE-6796:
--------------------------------------

Are you -1'ing this, Ted?
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489914#comment-13489914 ] 

Lars Hofhansl commented on HBASE-6796:
--------------------------------------

TestHFileCleaner.testHFileCleaning has been failing in every build since the checkin.
We have three options:
# fix the test
# disable the test
# revert the change

I'd be reluctant to do #3. At the same I cannot release the next RC with a consistently failing test.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488931#comment-13488931 ] 

Lars Hofhansl commented on HBASE-6796:
--------------------------------------

Oh I see, you did that with all the related changes. Fine then.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456330#comment-13456330 ] 

stack commented on HBASE-6796:
------------------------------

@Lars You fellas need this in 0.94?
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489936#comment-13489936 ] 

Lars Hofhansl commented on HBASE-6796:
--------------------------------------

Test didn't fail in latest build.
If a test is killed by a timeout, does the resource checker do the right thing?
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467456#comment-13467456 ] 

Ted Yu commented on HBASE-6796:
-------------------------------

There're 3 related JIRAs open at this point which may have pending changes in code code.

FYI
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lars Hofhansl updated HBASE-6796:
---------------------------------

    Description: See HBASE-5547
    
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Jesse Yates (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489051#comment-13489051 ] 

Jesse Yates commented on HBASE-6796:
------------------------------------

Thanks Lars!
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489918#comment-13489918 ] 

Lars Hofhansl commented on HBASE-6796:
--------------------------------------

Why is that test marked with a 2000ms timeout? Seems a bit short, it's doing a bunch of file manipulation. I am going to remove that annotation in favor of the default as an addendum.

                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456308#comment-13456308 ] 

Lars Hofhansl commented on HBASE-6796:
--------------------------------------

If there are concerns about this being backported into 0.94, let's discuss here.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Jesse Yates (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jesse Yates updated HBASE-6796:
-------------------------------

    Attachment: hbase-6796-v1.patch

Updating patch to include fix from HBASE-7082.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490398#comment-13490398 ] 

Hudson commented on HBASE-6796:
-------------------------------

Integrated in HBase-0.94-security-on-Hadoop-23 #9 (See [https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/9/])
    HBASE-6796 ADDENDUM, remove spurious time limit from testHFileCleaning (Revision 1405275)
HBASE-6796 Backport HBASE-5547, Don't delete HFiles in backup mode. (Jesse Yates) (Revision 1404762)

     Result = FAILURE
larsh : 
Files : 
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java

larsh : 
Files : 
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/BaseConfigurable.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/Chore.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/backup
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/LogCleaner.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/LogCleanerDelegate.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/TimeToLiveLogCleaner.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveHFileCleaner.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveLogCleaner.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
* /hbase/branches/0.94/src/main/resources/hbase-default.xml
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/backup
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestLogsCleaner.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/cleaner
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestCleanerChore.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestLogsCleaner.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/CheckedArchivingHFileCleaner.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHFileArchiveUtil.java

                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488993#comment-13488993 ] 

Lars Hofhansl commented on HBASE-6796:
--------------------------------------

I'm actually getting a failure with TestFSTableDescriptors :(
{code}
testCreateAndUpdate(org.apache.hadoop.hbase.util.TestFSTableDescriptors)  Time elapsed: 0.025 sec  <<< FAILURE!
java.lang.AssertionError: statuses.length=5
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.hadoop.hbase.util.TestFSTableDescriptors.testCreateAndUpdate(TestFSTableDescriptors.java:63)
{code}
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch, hbase-6796-v2.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Jesse Yates (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488939#comment-13488939 ] 

Jesse Yates commented on HBASE-6796:
------------------------------------

hmmm, I'll try a rebase and see.
                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch, hbase-6796-v1.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456585#comment-13456585 ] 

Lars Hofhansl commented on HBASE-6796:
--------------------------------------

Thanks Stack and Ted. Good discussion.

I'll commit Salesforce to the following actions:
* Address the three issues mentioned above (HBASE-6439, HBASE-6797, HBASE-6707)
* Run with this patch on our test cluster and poke some load at it, delete some large tables, etc. (so that it leads to lots of deleted HFiles)

                
> Backport HBASE-5547, Don't delete HFiles in backup mode.
> --------------------------------------------------------
>
>                 Key: HBASE-6796
>                 URL: https://issues.apache.org/jira/browse/HBASE-6796
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>             Fix For: 0.94.3
>
>         Attachments: hbase-5547-0.94-backport-v0.patch
>
>
> See HBASE-5547

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira