You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by "Riesland, Zack" <Za...@sensus.com> on 2015/07/06 16:12:51 UTC
Permissions Question
I've been running CsvBulkLoader as 'hbase' and that has worked well.
But I now need to integrate with some scripts that will be run as another user.
When I run under a different account, the CsvBulkLoader runs and creates the HFiles, but then encounters permission issues attempting to write the data to HBase.
Can someone point me in the right direction for solving this?
How can I give 'hbase' write permissions to a different user?
Thanks!
RE: Permissions Question
Posted by "Riesland, Zack" <Za...@sensus.com>.
Gabriel,
This is EXACTLY what I needed. Thanks!
From: Gabriel Reid [mailto:gabriel.reid@gmail.com]
Sent: Wednesday, July 08, 2015 1:40 AM
To: user@phoenix.apache.org
Subject: Re: Permissions Question
Hi Zack,
There are two options that I know of, and I think that both of them should work.
First is that you can supply a custom output directory to the bulk loader using the -o parameter (see http://phoenix.apache.org/bulk_dataload.html). In this way you can ensure that the output directory doesn't automatically change every time you run the jar.
The second is that you should be able to supply the parameter -Dfs.permissions.umask-mode=000 to the bulk load tool (before any other parameters). This manipulates the umask with which the files will be written, making the output readable and writable by everyone (which then allows hbase to move it under its own directory structure).
Assuming that at least one of these works for you (or even if they don't), could you add a ticket in the Phoenix JIRA (https://issues.apache.org/jira/browse/PHOENIX) so that we can track getting a more structural fix for this issue?
- Gabriel
On Tue, Jul 7, 2015 at 4:53 PM Riesland, Zack <Za...@sensus.com>> wrote:
Thanks Krishna,
The hfiles are stored in, for example, /tmp/daa6119d-f49e-485e-a6fe-1405d9c3f2a4/<structure based on table name>
‘tmp’ is owned by ‘hdfs’ in group ‘hdfs’.
‘daa6119d-f49e-485e-a6fe-1405d9c3f2a4’ is owned by my script user (‘user1’ for example) in group ‘hdfs’.
I cannot run the script as ‘hbase’, and the name of the folder (‘daa6119d-f49e-485e-a6fe-1405d9c3f2a4’ in this case) will change each time I run the jar, so explicitly doing a chown on that folder won’t help.
Do you know what change I need to make to ‘user1’ so that hfiles created by him will write to hbase?
From: Krishna [mailto:research800@gmail.com<ma...@gmail.com>]
Sent: Monday, July 06, 2015 3:11 PM
To: user@phoenix.apache.org<ma...@phoenix.apache.org>
Subject: Re: Permissions Question
The owner of the directory containing HFiles should be 'hbase' user and ownership can set using 'chown' command.
On Mon, Jul 6, 2015 at 7:12 AM, Riesland, Zack <Za...@sensus.com>> wrote:
I’ve been running CsvBulkLoader as ‘hbase’ and that has worked well.
But I now need to integrate with some scripts that will be run as another user.
When I run under a different account, the CsvBulkLoader runs and creates the HFiles, but then encounters permission issues attempting to write the data to HBase.
Can someone point me in the right direction for solving this?
How can I give ‘hbase’ write permissions to a different user?
Thanks!
Re: Permissions Question
Posted by Gabriel Reid <ga...@gmail.com>.
Hi Zack,
There are two options that I know of, and I think that both of them should
work.
First is that you can supply a custom output directory to the bulk loader
using the -o parameter (see http://phoenix.apache.org/bulk_dataload.html).
In this way you can ensure that the output directory doesn't automatically
change every time you run the jar.
The second is that you should be able to supply the parameter
-Dfs.permissions.umask-mode=000 to the bulk load tool (before any other
parameters). This manipulates the umask with which the files will be
written, making the output readable and writable by everyone (which then
allows hbase to move it under its own directory structure).
Assuming that at least one of these works for you (or even if they don't),
could you add a ticket in the Phoenix JIRA (
https://issues.apache.org/jira/browse/PHOENIX) so that we can track getting
a more structural fix for this issue?
- Gabriel
On Tue, Jul 7, 2015 at 4:53 PM Riesland, Zack <Za...@sensus.com>
wrote:
> Thanks Krishna,
>
>
>
> The hfiles are stored in, for example,
> /tmp/daa6119d-f49e-485e-a6fe-1405d9c3f2a4/<structure based on table name>
>
>
>
> ‘tmp’ is owned by ‘hdfs’ in group ‘hdfs’.
>
>
>
> ‘daa6119d-f49e-485e-a6fe-1405d9c3f2a4’ is owned by my script user (‘user1’
> for example) in group ‘hdfs’.
>
>
>
> I cannot run the script as ‘hbase’, and the name of the folder
> (‘daa6119d-f49e-485e-a6fe-1405d9c3f2a4’ in this case) will change each time
> I run the jar, so explicitly doing a chown on that folder won’t help.
>
>
>
> Do you know what change I need to make to ‘user1’ so that hfiles created
> by him will write to hbase?
>
>
>
> *From:* Krishna [mailto:research800@gmail.com]
> *Sent:* Monday, July 06, 2015 3:11 PM
> *To:* user@phoenix.apache.org
> *Subject:* Re: Permissions Question
>
>
>
> The owner of the directory containing HFiles should be 'hbase' user and
> ownership can set using 'chown' command.
>
>
>
> On Mon, Jul 6, 2015 at 7:12 AM, Riesland, Zack <Za...@sensus.com>
> wrote:
>
> I’ve been running CsvBulkLoader as ‘hbase’ and that has worked well.
>
>
>
> But I now need to integrate with some scripts that will be run as another
> user.
>
>
>
> When I run under a different account, the CsvBulkLoader runs and creates
> the HFiles, but then encounters permission issues attempting to write the
> data to HBase.
>
>
>
> Can someone point me in the right direction for solving this?
>
>
>
> How can I give ‘hbase’ write permissions to a different user?
>
>
>
> Thanks!
>
>
>
>
>
>
>
RE: Permissions Question
Posted by "Riesland, Zack" <Za...@sensus.com>.
Thanks Krishna,
The hfiles are stored in, for example, /tmp/daa6119d-f49e-485e-a6fe-1405d9c3f2a4/<structure based on table name>
‘tmp’ is owned by ‘hdfs’ in group ‘hdfs’.
‘daa6119d-f49e-485e-a6fe-1405d9c3f2a4’ is owned by my script user (‘user1’ for example) in group ‘hdfs’.
I cannot run the script as ‘hbase’, and the name of the folder (‘daa6119d-f49e-485e-a6fe-1405d9c3f2a4’ in this case) will change each time I run the jar, so explicitly doing a chown on that folder won’t help.
Do you know what change I need to make to ‘user1’ so that hfiles created by him will write to hbase?
From: Krishna [mailto:research800@gmail.com]
Sent: Monday, July 06, 2015 3:11 PM
To: user@phoenix.apache.org
Subject: Re: Permissions Question
The owner of the directory containing HFiles should be 'hbase' user and ownership can set using 'chown' command.
On Mon, Jul 6, 2015 at 7:12 AM, Riesland, Zack <Za...@sensus.com>> wrote:
I’ve been running CsvBulkLoader as ‘hbase’ and that has worked well.
But I now need to integrate with some scripts that will be run as another user.
When I run under a different account, the CsvBulkLoader runs and creates the HFiles, but then encounters permission issues attempting to write the data to HBase.
Can someone point me in the right direction for solving this?
How can I give ‘hbase’ write permissions to a different user?
Thanks!
Re: Permissions Question
Posted by Krishna <re...@gmail.com>.
The owner of the directory containing HFiles should be 'hbase' user and
ownership can set using 'chown' command.
On Mon, Jul 6, 2015 at 7:12 AM, Riesland, Zack <Za...@sensus.com>
wrote:
> I’ve been running CsvBulkLoader as ‘hbase’ and that has worked well.
>
>
>
> But I now need to integrate with some scripts that will be run as another
> user.
>
>
>
> When I run under a different account, the CsvBulkLoader runs and creates
> the HFiles, but then encounters permission issues attempting to write the
> data to HBase.
>
>
>
> Can someone point me in the right direction for solving this?
>
>
>
> How can I give ‘hbase’ write permissions to a different user?
>
>
>
> Thanks!
>
>
>
>
>