You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Lohit Vijayarenu (JIRA)" <ji...@apache.org> on 2011/08/05 20:11:27 UTC
[jira] [Created] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
FSUtils LeaseRecovery for non HDFS FileSystems.
-----------------------------------------------
Key: HBASE-4169
URL: https://issues.apache.org/jira/browse/HBASE-4169
Project: HBase
Issue Type: Bug
Components: util
Affects Versions: 0.90.3, 0.90.4
Reporter: Lohit Vijayarenu
FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081344#comment-13081344 ]
Hudson commented on HBASE-4169:
-------------------------------
Integrated in HBase-TRUNK #2095 (See [https://builds.apache.org/job/HBase-TRUNK/2095/])
HBASE-4169 FSUtils LeaseRecovery for non HDFS FileSystems -- reverted until we address issues raised around class name
stack :
Files :
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSNonHDFSUtils.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Assignee: Lohit Vijayarenu
> Attachments: 4169-v4.txt, 4169-v5.txt, HBASE-4169.1.patch, HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080303#comment-13080303 ]
Andrew Purtell commented on HBASE-4169:
---------------------------------------
I'm inclined to close this as invalid. It works with HDFS, which ships with Apache Hadoop, and we are Apache HBase that runs on Apache Hadoop. Is it incumbent on us to deal with the vagaries of XYZ vendor filesystems?
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081470#comment-13081470 ]
Hudson commented on HBASE-4169:
-------------------------------
Integrated in HBase-TRUNK #2097 (See [https://builds.apache.org/job/HBase-TRUNK/2097/])
HBASE-4169 FSUtils LeaseRecovery for non HDFS FileSystems
stack :
Files :
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Assignee: Lohit Vijayarenu
> Fix For: 0.92.0
>
> Attachments: 4169-v4.txt, 4169-v5.txt, HBASE-4169.1.patch, HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080316#comment-13080316 ]
Todd Lipcon commented on HBASE-4169:
------------------------------------
Abstractions suck. I think most systems programmers agree. But they're better than the alternative :)
That's why we have things like ioctl/fcntl, right? I wouldn't fault FileSystem here.
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081310#comment-13081310 ]
Andrew Purtell commented on HBASE-4169:
---------------------------------------
I agree with Todd. "NonHDFS" is not the right name for something that only works on MapR. Call it "MapRFS".
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Assignee: Lohit Vijayarenu
> Attachments: 4169-v4.txt, HBASE-4169.1.patch, HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lohit Vijayarenu updated HBASE-4169:
------------------------------------
Attachment: 4169-correction.txt
Please could we patch this. I have tested ths internally.
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Assignee: Lohit Vijayarenu
> Fix For: 0.92.0
>
> Attachments: 4169-correction.txt, 4169-v4.txt, 4169-v5.txt, HBASE-4169.1.patch, HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081227#comment-13081227 ]
Todd Lipcon commented on HBASE-4169:
------------------------------------
The non-HDFS one doesn't make much sense to me... I tried the chmod trick on our NFS filer here and it doesn't work -- opened a file from one node for write, and was able to continue to write/fsync even after I chmodded on a different node. It might be that there are some specific file systems where this works, but it's far from the norm in my experience.
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Attachments: HBASE-4169.1.patch, HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4169:
-------------------------
Resolution: Fixed
Fix Version/s: 0.92.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed to TRUNK. Thanks for the patch Lohit.
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Assignee: Lohit Vijayarenu
> Fix For: 0.92.0
>
> Attachments: 4169-v4.txt, 4169-v5.txt, HBASE-4169.1.patch, HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081312#comment-13081312 ]
Hudson commented on HBASE-4169:
-------------------------------
Integrated in HBase-TRUNK #2094 (See [https://builds.apache.org/job/HBase-TRUNK/2094/])
HBASE-4169 FSUtils LeaseRecovery for non HDFS FileSystems
stack :
Files :
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSNonHDFSUtils.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Assignee: Lohit Vijayarenu
> Attachments: 4169-v4.txt, HBASE-4169.1.patch, HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080469#comment-13080469 ]
stack commented on HBASE-4169:
------------------------------
@Lohit Patch is not bad. You did it this way because the bulk of FSUtils is static so not subclassable? The naming is a little awkward I'd say. FSImplementedUtils is a little odd when we still have FSUtils (And FSImplementedUtils is a factory so why not call it so?). I'd imagine going forward there'll be more than just this one method different across fs implementations? Do you think this method even belongs in our util package? I know thats where it was before you went messing but it seems more fundamental than a utility. Perhaps we need an fs package and in there you'd have an fs factory class that provides different fs implementations for different methods? You are trying to avoid doing 'mapr://' for the fs? (Even if you did mapr:// you'd need to change how we do this recoverLease). Good stuff.
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Attachments: HBASE-4169.1.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lohit Vijayarenu updated HBASE-4169:
------------------------------------
Status: Patch Available (was: Open)
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Attachments: HBASE-4169.1.patch, HBASE-4169.2.patch, HBASE-4196.3.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lohit Vijayarenu updated HBASE-4169:
------------------------------------
Attachment: HBASE-4196.3.v2.patch
I made little modification to previous patch. Included NonHDFS implementation of FSUtils which does chmod. If this patch is included, while running on non hdfs, users could just change their config to point to the NonHDFS class.
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Attachments: HBASE-4169.1.patch, HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081419#comment-13081419 ]
Lohit Vijayarenu commented on HBASE-4169:
-----------------------------------------
Thanks everyone for the help
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Assignee: Lohit Vijayarenu
> Fix For: 0.92.0
>
> Attachments: 4169-v4.txt, 4169-v5.txt, HBASE-4169.1.patch, HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081058#comment-13081058 ]
Todd Lipcon commented on HBASE-4169:
------------------------------------
This patch is a little strange - why use reflection here to access the static method in FSHDFSUtils instead of just creating a new interface or abstract class, and using non-static methods?
I think a more conventional approach would be:
- Create an abstract class FSUtils, which declares either default implementations or abstract methods for all of the things that different file systems would want to override
- This class would also have a method getInstance(Filesystem, Configuration), which returns a subclass of FSUtils dependent on scheme, using ReflectionUtils.newInstance
- If this ever becomes a performance hotspot we could add a cache here, but for rare operations like recoverLease it shouldn't matter
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Attachments: HBASE-4169.1.patch, HBASE-4169.2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-4169.
--------------------------
Resolution: Fixed
I applied the correction. Closing. Let me know if I have it wrong Lohit.
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Assignee: Lohit Vijayarenu
> Fix For: 0.92.0
>
> Attachments: 4169-correction.txt, 4169-v4.txt, 4169-v5.txt, HBASE-4169.1.patch, HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081787#comment-13081787 ]
stack commented on HBASE-4169:
------------------------------
I applied 4169-correction.txt. Thanks Lohit (You might want to make a doc patch to go along w/ this change Lohit to explain whats needed to make hbase run on mapr).
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Assignee: Lohit Vijayarenu
> Fix For: 0.92.0
>
> Attachments: 4169-correction.txt, 4169-v4.txt, 4169-v5.txt, HBASE-4169.1.patch, HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080761#comment-13080761 ]
Lohit Vijayarenu commented on HBASE-4169:
-----------------------------------------
Ran full test and I see few failures which are present even in trunk without the changes.
No new test failures due to this patch. Any suggestions how we should proceed.
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Attachments: HBASE-4169.1.patch, HBASE-4169.2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lohit Vijayarenu updated HBASE-4169:
------------------------------------
Attachment: HBASE-4169.1.patch
Attached is first version of patch. Please could anyone review this. Changes done are
1. Moved recoverLease to FSHDFSUtils
2. Added FSImplementedUtils which loads FSUtils based on config hbase.fsutils.fs.impl
3. Added FSHDFSUtils as default in hbase-default.xml
4. Changed all places where this is invoked.
I rand TestHLog which invokes this piece of code to make sure FSHDFSUtils is loaded and recover the lease
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Attachments: HBASE-4169.1.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081398#comment-13081398 ]
Lohit Vijayarenu commented on HBASE-4169:
-----------------------------------------
@stack if the class is in hbase core the users could run different versions of hbase without upgrading MapR.
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Assignee: Lohit Vijayarenu
> Attachments: 4169-v4.txt, 4169-v5.txt, HBASE-4169.1.patch, HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack reassigned HBASE-4169:
----------------------------
Assignee: Lohit Vijayarenu
Assigning Lohit (Added you as contributor Lohit)
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Assignee: Lohit Vijayarenu
> Attachments: 4169-v4.txt, HBASE-4169.1.patch, HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lohit Vijayarenu updated HBASE-4169:
------------------------------------
Status: Patch Available (was: Open)
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Attachments: HBASE-4169.1.patch, HBASE-4169.2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lohit Vijayarenu updated HBASE-4169:
------------------------------------
Attachment: HBASE-4169.2.patch
Attached is patch with changes incorporating comments from Stack.
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Attachments: HBASE-4169.1.patch, HBASE-4169.2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080467#comment-13080467 ]
stack commented on HBASE-4169:
------------------------------
@Andrew
bq. I'm inclined to close this as invalid. It works with HDFS, which ships with Apache Hadoop, and we are Apache HBase that runs on Apache Hadoop. Is it incumbent on us to deal with the vagaries of XYZ vendor filesystems?
I don't have a problem making "small" changes to accomodate other FS implementations. It could broaden our user base and experience gained atop other FSs could help improve the home base.
@Lohit
I don't think this will work:
{code}
if (!(fs instanceof DistributedFileSystem)) {
// Change permission on file to ReadOnly.
return
}
{code}
The above will happen when running atop the local filesystem, not just for mapr (or ceph).
I think the Todd suggestion of a mapr:// would be a better way to go though as the lads note above, the abstraction is likely lacking when it comes to other FS implementations (let me look at your patch).
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Attachments: HBASE-4169.1.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081253#comment-13081253 ]
Lohit Vijayarenu commented on HBASE-4169:
-----------------------------------------
@Todd Yes that is the reason it is optional. Having with distribution would help users use vanila HBase on non HDFS without having them include file.
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Attachments: HBASE-4169.1.patch, HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lohit Vijayarenu reopened HBASE-4169:
-------------------------------------
Sorry guys. In hurry I ended up adding wrong FSMapRUtils. I will update patch soon against trunk. Please could we include this.
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Assignee: Lohit Vijayarenu
> Fix For: 0.92.0
>
> Attachments: 4169-v4.txt, 4169-v5.txt, HBASE-4169.1.patch, HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lohit Vijayarenu updated HBASE-4169:
------------------------------------
Attachment: HBASE-4196.3.patch
Attached is updated patch incorporating Todd's comments. I ran tests and do not see any new tests fail.
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Attachments: HBASE-4169.1.patch, HBASE-4169.2.patch, HBASE-4196.3.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081307#comment-13081307 ]
Todd Lipcon commented on HBASE-4169:
------------------------------------
I am still -1 on having this "generic interface" since it works on only one filesystem I've ever heard of. It certainly doesn't work on local filesystem either. If we're going to implement something for mapr, we should just call it Mapr.
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Assignee: Lohit Vijayarenu
> Attachments: 4169-v4.txt, HBASE-4169.1.patch, HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080472#comment-13080472 ]
Lohit Vijayarenu commented on HBASE-4169:
-----------------------------------------
@Stack Thanks for reviewing the patch. Yes, I made this separate class because FSUtils already has many other methods which works across non HDFS, so I moved recoverFileLease alone to different class. Are you suggesting I move FSImplementedUtils and FSHDFSUtils to {{org.apache.hadoop.hbase.fs.util}}
Yes I was trying to avoid mapfs://. We have maprfs jar which includes client. I would planning to drop MapRFS implementation of recoverFileLease which would do chmod and update hbase-site.xml when running under maprfs. Would this work?
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Attachments: HBASE-4169.1.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lohit Vijayarenu updated HBASE-4169:
------------------------------------
Attachment: 4169-v5.txt
Attached is patch based on Stack's changes (4169-v4.txt). Changed name of NonHDFS to MapR
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Assignee: Lohit Vijayarenu
> Attachments: 4169-v4.txt, 4169-v5.txt, HBASE-4169.1.patch, HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lohit Vijayarenu updated HBASE-4169:
------------------------------------
Status: Open (was: Patch Available)
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Attachments: HBASE-4169.1.patch, HBASE-4169.2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080346#comment-13080346 ]
Lohit Vijayarenu commented on HBASE-4169:
-----------------------------------------
Thanks Todd. I will make a patch to load current FSUtils::recoverFileLease if implementation is hdfs. Users who want to run on different filesystem can change config to load different implementation.
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080314#comment-13080314 ]
Milind Bhandarkar commented on HBASE-4169:
------------------------------------------
This suggests deeper flaws in the the FileSystem abstraction in Hadoop. If one has to do:
{code}
if (fs instanceof DistributedFileSystem) {
// hdfs-specific functionality
}
{code}
then it defeats the purpose of having o.a.h.fs as public apis and o.a.h.hdfs as implementation.
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081387#comment-13081387 ]
stack commented on HBASE-4169:
------------------------------
@Lohit Does the MapR class need to be in hbase? Could it be in jars of yours rather than in hbase core? If it has to be in hbase core, no problem, will commit as is.
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Assignee: Lohit Vijayarenu
> Attachments: 4169-v4.txt, 4169-v5.txt, HBASE-4169.1.patch, HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081314#comment-13081314 ]
stack commented on HBASE-4169:
------------------------------
Reverting to fix class name (please base new patch on mine Lohit)
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Assignee: Lohit Vijayarenu
> Attachments: 4169-v4.txt, HBASE-4169.1.patch, HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080306#comment-13080306 ]
Todd Lipcon commented on HBASE-4169:
------------------------------------
A design I"d be OK with us to use a set of configurations like {{hbase.fsutil.<scheme>.impl}} to define a particular class for each filesystem. For example, {{hbase.fsutil.hdfs.impl}} would instantiate the FSUtils class we have right now. Anyone with another filesystem could drop in their own with a configuration, without having to edit source upstream.
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080116#comment-13080116 ]
Lohit Vijayarenu commented on HBASE-4169:
-----------------------------------------
One suggestion is to do something like.
{code}
if (!(fs instanceof DistributedFileSystem)) {
// Change permission on file to ReadOnly.
return
}
// HDFS code to deal with lease recover as of today.
{code}
Will this work?
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080309#comment-13080309 ]
Andrew Purtell commented on HBASE-4169:
---------------------------------------
bq. A design I"d be OK with us to use a set of configurations like hbase.fsutil.<scheme>.impl to define a particular class for each filesystem.
That's a good idea Todd, +1. Lohit?
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4169) FSUtils LeaseRecovery for non HDFS
FileSystems.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4169:
-------------------------
Attachment: 4169-v4.txt
Here is what I committed to trunk. Its Lohit's patch some doc including note that the nonhdfs util class is just a generic implementation that may not work (doesn't work on nfs for example as per todd). Thanks for the patch Lohit.
> FSUtils LeaseRecovery for non HDFS FileSystems.
> -----------------------------------------------
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.90.3, 0.90.4
> Reporter: Lohit Vijayarenu
> Attachments: 4169-v4.txt, HBASE-4169.1.patch, HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before splitting hlog file.
> This might not work for other filesystem implementations.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira