You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Sean Bigdatafun <se...@gmail.com> on 2012/08/16 07:50:31 UTC
Hive directory permissions
I ran the following command trying to setup my hive directory, but got the
following error.
root@cdh-client:/# hadoop fs -mkdir /user/hive/warehouse
mkdir: org.apache.hadoop.security.AccessControlException: Permission
denied: user=root, access=WRITE, inode="/user":hdfs:hadoop:drwxr-xr-x
Can someone tell me what is the "correct" ownership for hive directories?
My "/user" directory is with the following ownership:
user<http://cdh-slave2-16g.hwhadoop.com:50075/browseDirectory.jsp?dir=%2Fuser&namenodeInfoPort=50070>
dir2012-08-14 16:51rwxr-xr-xhdfshadoop
Should I create a sub-directory under user with the following permission?
h<http://cdh-slave2-16g.hwhadoop.com:50075/browseDirectory.jsp?dir=%2Fuser&namenodeInfoPort=50070>
ivedir2012-08-14 16:51rwxr-xr-xhivehadoop
Thanks,
Sean
Re: Hive directory permissions
Posted by John Meagher <jo...@gmail.com>.
Creating the /user/hive/warehouse folder is a one-time setup step that can
be done as the hdfs user. With g+w permissions any user can then create
and read the tables.
On Thu, Aug 16, 2012 at 9:57 AM, Connell, Chuck <Ch...@nuance.com>wrote:
> I have no doubt that works, but surely a Hive user should not need sudo
> privileges! I am also looking for best practices, since we have run into
> the same.****
>
> ** **
>
> ** **
>
> *From:* Himanish Kushary [mailto:himanish@gmail.com]
> *Sent:* Thursday, August 16, 2012 9:51 AM
>
> *To:* user@hive.apache.org
> *Subject:* Re: Hive directory permissions****
>
> ** **
>
> We usually start the shell thru sudo,otherwise we get a "Permission
> denied" while creating Hive tables.****
>
> ** **
>
> But this is a good point, any suggestions/best practices from the user
> community ?****
>
> ** **
>
> Thanks****
>
> On Thu, Aug 16, 2012 at 9:37 AM, Connell, Chuck <Ch...@nuance.com>
> wrote:****
>
> I have run into similar problems. Thanks for the suggestions. One concern…
> Isn’t hdfs a highly privileged user within the Hadoop cluster? So do we
> really want it to be standard practice for all Hive users to su to hdfs?**
> **
>
> ****
>
> Chuck Connell****
>
> Nuance R&D Data Team****
>
> Burlington, MA****
>
> ****
>
> *From:* Himanish Kushary [mailto:himanish@gmail.com]
> *Sent:* Thursday, August 16, 2012 9:30 AM
> *To:* user@hive.apache.org
> *Subject:* Re: Hive directory permissions****
>
> ****
>
> Hi Sean,****
>
> ****
>
> From the Hive language manual - "Moreover, we strongly advise users to
> create the HDFS directories /tmp and /user/hive/warehouse
> (aka hive.metastore.warehouse.dir) and set them chmod g+w before tables
> are created in Hive"****
>
> ****
>
> My warehouse directory has the following permissions:****
>
> ****
>
> *Name*****
>
> *Type*****
>
> *Size*****
>
> *Replication*****
>
> *Block Size*****
>
> *Modification Time*****
>
> *Permission*****
>
> *Owner*****
>
> *Group*****
>
> *warehouse<http://h-node02:50075/browseDirectory.jsp?dir=%2Fuser%2Fhive%2Fwarehouse&namenodeInfoPort=50070&nnaddr=hdfs01:8020>
> *****
>
> *dir*****
>
> *2012-08-15 15:51*****
>
> *rwxrwxr-x*****
>
> *hdfs*****
>
> *supergroup*****
>
>
> Also I make sure that I run the hive CLI as 'hdfs' user.****
>
> ****
>
> Hope this helps****
>
> ****
>
> Thanks****
>
> Himanish****
>
> On Thu, Aug 16, 2012 at 1:50 AM, Sean Bigdatafun <
> sean.bigdatafun@gmail.com> wrote:****
>
> I ran the following command trying to setup my hive directory, but got the
> following error. ****
>
> ****
>
> root@cdh-client:/# hadoop fs -mkdir /user/hive/warehouse****
>
> mkdir: org.apache.hadoop.security.AccessControlException: Permission
> denied: user=root, access=WRITE, inode="/user":hdfs:hadoop:drwxr-xr-x****
>
> ****
>
> Can someone tell me what is the "correct" ownership for hive directories?
> My "/user" directory is with the following ownership: ****
>
> user<http://cdh-slave2-16g.hwhadoop.com:50075/browseDirectory.jsp?dir=%2Fuser&namenodeInfoPort=50070>
> ****
>
> dir****
>
> 2012-08-14 16:51****
>
> rwxr-xr-x****
>
> hdfs****
>
> hadoop****
>
> ****
>
> Should I create a sub-directory under user with the following permission?*
> ***
>
> h<http://cdh-slave2-16g.hwhadoop.com:50075/browseDirectory.jsp?dir=%2Fuser&namenodeInfoPort=50070>
> ive****
>
> dir****
>
> 2012-08-14 16:51****
>
> rwxr-xr-x****
>
> hive****
>
> hadoop****
>
>
> Thanks,****
>
> Sean****
>
> ****
>
> ****
>
> ****
>
>
>
> ****
>
> ****
>
> --
> Thanks & Regards
> Himanish****
>
>
>
> ****
>
> ** **
>
> --
> Thanks & Regards
> Himanish****
>
RE: Hive directory permissions
Posted by "Connell, Chuck" <Ch...@nuance.com>.
I have no doubt that works, but surely a Hive user should not need sudo privileges! I am also looking for best practices, since we have run into the same.
From: Himanish Kushary [mailto:himanish@gmail.com]
Sent: Thursday, August 16, 2012 9:51 AM
To: user@hive.apache.org
Subject: Re: Hive directory permissions
We usually start the shell thru sudo,otherwise we get a "Permission denied" while creating Hive tables.
But this is a good point, any suggestions/best practices from the user community ?
Thanks
On Thu, Aug 16, 2012 at 9:37 AM, Connell, Chuck <Ch...@nuance.com>> wrote:
I have run into similar problems. Thanks for the suggestions. One concern... Isn't hdfs a highly privileged user within the Hadoop cluster? So do we really want it to be standard practice for all Hive users to su to hdfs?
Chuck Connell
Nuance R&D Data Team
Burlington, MA
From: Himanish Kushary [mailto:himanish@gmail.com<ma...@gmail.com>]
Sent: Thursday, August 16, 2012 9:30 AM
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: Hive directory permissions
Hi Sean,
>From the Hive language manual - "Moreover, we strongly advise users to create the HDFS directories /tmp and /user/hive/warehouse
(aka hive.metastore.warehouse.dir) and set them chmod g+w before tables are created in Hive"
My warehouse directory has the following permissions:
Name
Type
Size
Replication
Block Size
Modification Time
Permission
Owner
Group
warehouse<http://h-node02:50075/browseDirectory.jsp?dir=%2Fuser%2Fhive%2Fwarehouse&namenodeInfoPort=50070&nnaddr=hdfs01:8020>
dir
2012-08-15 15:51
rwxrwxr-x
hdfs
supergroup
Also I make sure that I run the hive CLI as 'hdfs' user.
Hope this helps
Thanks
Himanish
On Thu, Aug 16, 2012 at 1:50 AM, Sean Bigdatafun <se...@gmail.com>> wrote:
I ran the following command trying to setup my hive directory, but got the following error.
root@cdh-client:/# hadoop fs -mkdir /user/hive/warehouse
mkdir: org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user":hdfs:hadoop:drwxr-xr-x
Can someone tell me what is the "correct" ownership for hive directories? My "/user" directory is with the following ownership:
user<http://cdh-slave2-16g.hwhadoop.com:50075/browseDirectory.jsp?dir=%2Fuser&namenodeInfoPort=50070>
dir
2012-08-14 16:51
rwxr-xr-x
hdfs
hadoop
Should I create a sub-directory under user with the following permission?
h<http://cdh-slave2-16g.hwhadoop.com:50075/browseDirectory.jsp?dir=%2Fuser&namenodeInfoPort=50070>ive
dir
2012-08-14 16:51
rwxr-xr-x
hive
hadoop
Thanks,
Sean
--
Thanks & Regards
Himanish
--
Thanks & Regards
Himanish
Re: Hive directory permissions
Posted by Himanish Kushary <hi...@gmail.com>.
We usually start the shell thru sudo,otherwise we get a "Permission denied"
while creating Hive tables.
But this is a good point, any suggestions/best practices from the user
community ?
Thanks
On Thu, Aug 16, 2012 at 9:37 AM, Connell, Chuck <Ch...@nuance.com>wrote:
> I have run into similar problems. Thanks for the suggestions. One
> concern… Isn’t hdfs a highly privileged user within the Hadoop cluster? So
> do we really want it to be standard practice for all Hive users to su to
> hdfs?****
>
> ** **
>
> Chuck Connell****
>
> Nuance R&D Data Team****
>
> Burlington, MA****
>
> ** **
>
> *From:* Himanish Kushary [mailto:himanish@gmail.com]
> *Sent:* Thursday, August 16, 2012 9:30 AM
> *To:* user@hive.apache.org
> *Subject:* Re: Hive directory permissions****
>
> ** **
>
> Hi Sean,****
>
> ** **
>
> From the Hive language manual - "Moreover, we strongly advise users to
> create the HDFS directories /tmp and /user/hive/warehouse
> (aka hive.metastore.warehouse.dir) and set them chmod g+w before tables
> are created in Hive"****
>
> ** **
>
> My warehouse directory has the following permissions:****
>
> ** **
>
> *Name*****
>
> *Type*****
>
> *Size*****
>
> *Replication*****
>
> *Block Size*****
>
> *Modification Time*****
>
> *Permission*****
>
> *Owner*****
>
> *Group*****
>
> *warehouse<http://h-node02:50075/browseDirectory.jsp?dir=%2Fuser%2Fhive%2Fwarehouse&namenodeInfoPort=50070&nnaddr=hdfs01:8020>
> *****
>
> *dir*****
>
> *2012-08-15 15:51*****
>
> *rwxrwxr-x*****
>
> *hdfs*****
>
> *supergroup*****
>
>
> Also I make sure that I run the hive CLI as 'hdfs' user.****
>
> ** **
>
> Hope this helps****
>
> ** **
>
> Thanks****
>
> Himanish****
>
> On Thu, Aug 16, 2012 at 1:50 AM, Sean Bigdatafun <
> sean.bigdatafun@gmail.com> wrote:****
>
> I ran the following command trying to setup my hive directory, but got the
> following error. ****
>
> ** **
>
> root@cdh-client:/# hadoop fs -mkdir /user/hive/warehouse****
>
> mkdir: org.apache.hadoop.security.AccessControlException: Permission
> denied: user=root, access=WRITE, inode="/user":hdfs:hadoop:drwxr-xr-x****
>
> ** **
>
> Can someone tell me what is the "correct" ownership for hive directories?
> My "/user" directory is with the following ownership: ****
>
> user<http://cdh-slave2-16g.hwhadoop.com:50075/browseDirectory.jsp?dir=%2Fuser&namenodeInfoPort=50070>
> ****
>
> dir****
>
> 2012-08-14 16:51****
>
> rwxr-xr-x****
>
> hdfs****
>
> hadoop****
>
> ** **
>
> Should I create a sub-directory under user with the following permission?*
> ***
>
> h<http://cdh-slave2-16g.hwhadoop.com:50075/browseDirectory.jsp?dir=%2Fuser&namenodeInfoPort=50070>
> ive****
>
> dir****
>
> 2012-08-14 16:51****
>
> rwxr-xr-x****
>
> hive****
>
> hadoop****
>
>
> Thanks,****
>
> Sean****
>
> ****
>
> ** **
>
> ** **
>
>
>
> ****
>
> ** **
>
> --
> Thanks & Regards
> Himanish****
>
--
Thanks & Regards
Himanish
RE: Hive directory permissions
Posted by "Connell, Chuck" <Ch...@nuance.com>.
I have run into similar problems. Thanks for the suggestions. One concern... Isn't hdfs a highly privileged user within the Hadoop cluster? So do we really want it to be standard practice for all Hive users to su to hdfs?
Chuck Connell
Nuance R&D Data Team
Burlington, MA
From: Himanish Kushary [mailto:himanish@gmail.com]
Sent: Thursday, August 16, 2012 9:30 AM
To: user@hive.apache.org
Subject: Re: Hive directory permissions
Hi Sean,
>From the Hive language manual - "Moreover, we strongly advise users to create the HDFS directories /tmp and /user/hive/warehouse
(aka hive.metastore.warehouse.dir) and set them chmod g+w before tables are created in Hive"
My warehouse directory has the following permissions:
Name
Type
Size
Replication
Block Size
Modification Time
Permission
Owner
Group
warehouse<http://h-node02:50075/browseDirectory.jsp?dir=%2Fuser%2Fhive%2Fwarehouse&namenodeInfoPort=50070&nnaddr=hdfs01:8020>
dir
2012-08-15 15:51
rwxrwxr-x
hdfs
supergroup
Also I make sure that I run the hive CLI as 'hdfs' user.
Hope this helps
Thanks
Himanish
On Thu, Aug 16, 2012 at 1:50 AM, Sean Bigdatafun <se...@gmail.com>> wrote:
I ran the following command trying to setup my hive directory, but got the following error.
root@cdh-client:/# hadoop fs -mkdir /user/hive/warehouse
mkdir: org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user":hdfs:hadoop:drwxr-xr-x
Can someone tell me what is the "correct" ownership for hive directories? My "/user" directory is with the following ownership:
user<http://cdh-slave2-16g.hwhadoop.com:50075/browseDirectory.jsp?dir=%2Fuser&namenodeInfoPort=50070>
dir
2012-08-14 16:51
rwxr-xr-x
hdfs
hadoop
Should I create a sub-directory under user with the following permission?
h<http://cdh-slave2-16g.hwhadoop.com:50075/browseDirectory.jsp?dir=%2Fuser&namenodeInfoPort=50070>ive
dir
2012-08-14 16:51
rwxr-xr-x
hive
hadoop
Thanks,
Sean
--
Thanks & Regards
Himanish
Re: Hive directory permissions
Posted by Himanish Kushary <hi...@gmail.com>.
Hi Sean,
>From the Hive language manual - "Moreover, we strongly advise users to
create the HDFS directories /tmp and /user/hive/warehouse
(aka hive.metastore.warehouse.dir) and set them chmod g+w before tables are
created in Hive"
My warehouse directory has the following permissions:
*Name*
*Type*
*Size*
*Replication*
*Block Size*
*Modification Time*
*Permission*
*Owner*
*Group*
*warehouse<http://h-node02:50075/browseDirectory.jsp?dir=%2Fuser%2Fhive%2Fwarehouse&namenodeInfoPort=50070&nnaddr=hdfs01:8020>
*
*dir*
**
**
**
*2012-08-15 15:51*
*rwxrwxr-x*
*hdfs*
*supergroup*
Also I make sure that I run the hive CLI as 'hdfs' user.
Hope this helps
Thanks
Himanish
On Thu, Aug 16, 2012 at 1:50 AM, Sean Bigdatafun
<se...@gmail.com>wrote:
> I ran the following command trying to setup my hive directory, but got the
> following error.
>
> root@cdh-client:/# hadoop fs -mkdir /user/hive/warehouse
> mkdir: org.apache.hadoop.security.AccessControlException: Permission
> denied: user=root, access=WRITE, inode="/user":hdfs:hadoop:drwxr-xr-x
>
> Can someone tell me what is the "correct" ownership for hive directories?
> My "/user" directory is with the following ownership:
> user<http://cdh-slave2-16g.hwhadoop.com:50075/browseDirectory.jsp?dir=%2Fuser&namenodeInfoPort=50070>
> dir 2012-08-14 16:51 rwxr-xr-xhdfs hadoop
> Should I create a sub-directory under user with the following permission?
> h<http://cdh-slave2-16g.hwhadoop.com:50075/browseDirectory.jsp?dir=%2Fuser&namenodeInfoPort=50070>
> ive dir 2012-08-14 16:51 rwxr-xr-xhive hadoop
> Thanks,
> Sean
>
>
>
>
--
Thanks & Regards
Himanish