You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jeff Lord (JIRA)" <ji...@apache.org> on 2012/07/18 21:13:34 UTC

[jira] [Created] (HBASE-6422) Add switch in LoadIncrementalHFiles API to allow for programatically changing

Jeff Lord created HBASE-6422:
--------------------------------

             Summary: Add switch in LoadIncrementalHFiles API to allow for programatically changing 
                 Key: HBASE-6422
                 URL: https://issues.apache.org/jira/browse/HBASE-6422
             Project: HBase
          Issue Type: New Feature
          Components: mapred
    Affects Versions: 0.94.0
            Reporter: Jeff Lord


Hbase bulk load often requires manual chown and permission changes.
Usually it goes something like try to run completebulkload and it fails with the following hdfs error:

org.apache.hadoop.security.AccessControlException:
org.apache.hadoop.security.AccessControlException:
Permission denied:  user=hbase, access=WRITE, inode="mydata":
hadoop:supergroup:rwxr-xr-x

To work around this mismatch between Hadoop and HBase user permissions, you can make both users share a group: that is, the user where you run the MapReduce jobs and the user running HBase. Then, after running your MapReduce job, you can chgrp the output directory to the HBase group, and run chmod g+w. This allows the bulk loader to move the files into the HBase data directory.

It would be useful if there was a way to do this in the LoadIncrementalHFiles API

It is the case of linux permissions too:
If we have:
file owner:me group:me
We can chown to group:x and owner:x, without needing special permissions
chown x:x file

Then when we trigger bulk load, HBase does the fs -mv, and finds that the owner is itself, so no permission hitches. Goes smooth.

We are thinking that a switch to turn this on/off (default off) would be nice to have.



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6422) Add switch in LoadIncrementalHFiles API to allow for programatically changing perms on output directory

Posted by "Jeff Lord (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-6422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jeff Lord updated HBASE-6422:
-----------------------------

    Summary: Add switch in LoadIncrementalHFiles API to allow for programatically changing perms on output directory  (was: Add switch in LoadIncrementalHFiles API to allow for programatically changing )
    
> Add switch in LoadIncrementalHFiles API to allow for programatically changing perms on output directory
> -------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-6422
>                 URL: https://issues.apache.org/jira/browse/HBASE-6422
>             Project: HBase
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.94.0
>            Reporter: Jeff Lord
>
> Hbase bulk load often requires manual chown and permission changes.
> Usually it goes something like try to run completebulkload and it fails with the following hdfs error:
> org.apache.hadoop.security.AccessControlException:
> org.apache.hadoop.security.AccessControlException:
> Permission denied:  user=hbase, access=WRITE, inode="mydata":
> hadoop:supergroup:rwxr-xr-x
> To work around this mismatch between Hadoop and HBase user permissions, you can make both users share a group: that is, the user where you run the MapReduce jobs and the user running HBase. Then, after running your MapReduce job, you can chgrp the output directory to the HBase group, and run chmod g+w. This allows the bulk loader to move the files into the HBase data directory.
> It would be useful if there was a way to do this in the LoadIncrementalHFiles API
> It is the case of linux permissions too:
> If we have:
> file owner:me group:me
> We can chown to group:x and owner:x, without needing special permissions
> chown x:x file
> Then when we trigger bulk load, HBase does the fs -mv, and finds that the owner is itself, so no permission hitches. Goes smooth.
> We are thinking that a switch to turn this on/off (default off) would be nice to have.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6422) Add switch in LoadIncrementalHFiles API to allow for programatically changing perms on output directory

Posted by "Harsh J (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467470#comment-13467470 ] 

Harsh J commented on HBASE-6422:
--------------------------------

HBASE-5498 should address this. Turns out we can't do a chown unless we're in the HDFS superuser group and/or are root in LFS.

Lets resolve this as a dupe of HBASE-5498.
                
> Add switch in LoadIncrementalHFiles API to allow for programatically changing perms on output directory
> -------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-6422
>                 URL: https://issues.apache.org/jira/browse/HBASE-6422
>             Project: HBase
>          Issue Type: New Feature
>    Affects Versions: 0.94.0
>            Reporter: Jeff Lord
>
> Hbase bulk load often requires manual chown and permission changes.
> Usually it goes something like try to run completebulkload and it fails with the following hdfs error:
> org.apache.hadoop.security.AccessControlException:
> org.apache.hadoop.security.AccessControlException:
> Permission denied:  user=hbase, access=WRITE, inode="mydata":
> hadoop:supergroup:rwxr-xr-x
> To work around this mismatch between Hadoop and HBase user permissions, you can make both users share a group: that is, the user where you run the MapReduce jobs and the user running HBase. Then, after running your MapReduce job, you can chgrp the output directory to the HBase group, and run chmod g+w. This allows the bulk loader to move the files into the HBase data directory.
> It would be useful if there was a way to do this in the LoadIncrementalHFiles API
> It is the case of linux permissions too:
> If we have:
> file owner:me group:me
> We can chown to group:x and owner:x, without needing special permissions
> chown x:x file
> Then when we trigger bulk load, HBase does the fs -mv, and finds that the owner is itself, so no permission hitches. Goes smooth.
> We are thinking that a switch to turn this on/off (default off) would be nice to have.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-6422) Add switch in LoadIncrementalHFiles API to allow for programatically changing perms on output directory

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-6422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack resolved HBASE-6422.
--------------------------

    Resolution: Duplicate

Dupe of HBASE-5498 (Harsh J says it is so!)
                
> Add switch in LoadIncrementalHFiles API to allow for programatically changing perms on output directory
> -------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-6422
>                 URL: https://issues.apache.org/jira/browse/HBASE-6422
>             Project: HBase
>          Issue Type: New Feature
>    Affects Versions: 0.94.0
>            Reporter: Jeff Lord
>
> Hbase bulk load often requires manual chown and permission changes.
> Usually it goes something like try to run completebulkload and it fails with the following hdfs error:
> org.apache.hadoop.security.AccessControlException:
> org.apache.hadoop.security.AccessControlException:
> Permission denied:  user=hbase, access=WRITE, inode="mydata":
> hadoop:supergroup:rwxr-xr-x
> To work around this mismatch between Hadoop and HBase user permissions, you can make both users share a group: that is, the user where you run the MapReduce jobs and the user running HBase. Then, after running your MapReduce job, you can chgrp the output directory to the HBase group, and run chmod g+w. This allows the bulk loader to move the files into the HBase data directory.
> It would be useful if there was a way to do this in the LoadIncrementalHFiles API
> It is the case of linux permissions too:
> If we have:
> file owner:me group:me
> We can chown to group:x and owner:x, without needing special permissions
> chown x:x file
> Then when we trigger bulk load, HBase does the fs -mv, and finds that the owner is itself, so no permission hitches. Goes smooth.
> We are thinking that a switch to turn this on/off (default off) would be nice to have.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira