You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Inosh Goonewardena <in...@gmail.com> on 2015/04/06 10:10:45 UTC
Re: File permission of Hadoop staging directory getting changed time
to time
Hi All,
Really appreciate any input on this issue.
Thanks and Regards,
Inosh
On Tue, Jan 20, 2015 at 10:29 PM, Inosh Goonewardena <in...@gmail.com>
wrote:
> Hi All,
>
> We have setup a Hadoop cluster using hadoop 1.0.4 and we use Hive to
> submit map/reduce jobs. But we have noticed that these map/reduce job
> submission failing time to time due to a permission issue occurring in
> Hadoop. When this happens, we have to use the hadoop hdfs command and
> change the file permission of mapreduce staging directory. Following is the
> error that is getting logged.
>
> [2015-01-16 05:30:13,314] ERROR {org.apache.hadoop.hive.ql.exec.Task} -
> Job Submission failed with exception 'java.io.IOException(The
> ownership/permissions on the staging directory hdfs://
> hadoop0.test.com:9000/mnt/hadoop_tmp/mapred/staging/inosh/.staging is not
> as expected. It is owned by inosh and permissions are rwxr-xr-x. The
> directory must be owned by the submitter inosh or by inosh and permissions
> must be rwx------)'
> java.io.IOException: The ownership/permissions on the staging directory
> hdfs://hadoop0.test.com:9000/mnt/hadoop_tmp/mapred/staging/inosh/.staging
> is not as expected. It is owned by inosh and permissions are rwxr-xr-x. The
> directory must be owned by the submitter inosh or by inosh and permissions
> must be rwx------
> at
> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:108)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:798)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:792)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1123)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:792)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:766)
> at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:460)
> at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:129)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:62)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1351)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1126)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:934)
> at
> org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:201)
> at
> org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:187)
> ...
>
>
> When we get this error we go to hadoop name node and execute following
> command to set proper permission to the folder mentioned in the exception.
> ./hadoop dfs -chmod 700 /mnt/hadoop_tmp/mapred/staging/inosh/.staging
>
>
> We would like know why this happens and how to fix this permanently.
>
>
> Thanks and Regards,
> Inosh
>
Re: File permission of Hadoop staging directory getting changed time
to time
Posted by Harshit Mathur <ma...@gmail.com>.
try running with sudo hive.
On Mon, Apr 6, 2015 at 1:40 PM, Inosh Goonewardena <in...@gmail.com>
wrote:
> Hi All,
>
> Really appreciate any input on this issue.
>
> Thanks and Regards,
> Inosh
>
> On Tue, Jan 20, 2015 at 10:29 PM, Inosh Goonewardena <in...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> We have setup a Hadoop cluster using hadoop 1.0.4 and we use Hive to
>> submit map/reduce jobs. But we have noticed that these map/reduce job
>> submission failing time to time due to a permission issue occurring in
>> Hadoop. When this happens, we have to use the hadoop hdfs command and
>> change the file permission of mapreduce staging directory. Following is the
>> error that is getting logged.
>>
>> [2015-01-16 05:30:13,314] ERROR {org.apache.hadoop.hive.ql.exec.Task} -
>> Job Submission failed with exception 'java.io.IOException(The
>> ownership/permissions on the staging directory hdfs://
>> hadoop0.test.com:9000/mnt/hadoop_tmp/mapred/staging/inosh/.staging is
>> not as expected. It is owned by inosh and permissions are rwxr-xr-x. The
>> directory must be owned by the submitter inosh or by inosh and permissions
>> must be rwx------)'
>> java.io.IOException: The ownership/permissions on the staging directory
>> hdfs://hadoop0.test.com:9000/mnt/hadoop_tmp/mapred/staging/inosh/.staging
>> is not as expected. It is owned by inosh and permissions are rwxr-xr-x. The
>> directory must be owned by the submitter inosh or by inosh and permissions
>> must be rwx------
>> at
>> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:108)
>> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:798)
>> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:792)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1123)
>> at
>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:792)
>> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:766)
>> at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:460)
>> at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136)
>> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:129)
>> at
>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:62)
>> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1351)
>> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1126)
>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:934)
>> at
>> org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:201)
>> at
>> org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:187)
>> ...
>>
>>
>> When we get this error we go to hadoop name node and execute following
>> command to set proper permission to the folder mentioned in the exception.
>> ./hadoop dfs -chmod 700 /mnt/hadoop_tmp/mapred/staging/inosh/.staging
>>
>>
>> We would like know why this happens and how to fix this permanently.
>>
>>
>> Thanks and Regards,
>> Inosh
>>
>
>
--
Harshit Mathur
Re: File permission of Hadoop staging directory getting changed time
to time
Posted by Harshit Mathur <ma...@gmail.com>.
try running with sudo hive.
On Mon, Apr 6, 2015 at 1:40 PM, Inosh Goonewardena <in...@gmail.com>
wrote:
> Hi All,
>
> Really appreciate any input on this issue.
>
> Thanks and Regards,
> Inosh
>
> On Tue, Jan 20, 2015 at 10:29 PM, Inosh Goonewardena <in...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> We have setup a Hadoop cluster using hadoop 1.0.4 and we use Hive to
>> submit map/reduce jobs. But we have noticed that these map/reduce job
>> submission failing time to time due to a permission issue occurring in
>> Hadoop. When this happens, we have to use the hadoop hdfs command and
>> change the file permission of mapreduce staging directory. Following is the
>> error that is getting logged.
>>
>> [2015-01-16 05:30:13,314] ERROR {org.apache.hadoop.hive.ql.exec.Task} -
>> Job Submission failed with exception 'java.io.IOException(The
>> ownership/permissions on the staging directory hdfs://
>> hadoop0.test.com:9000/mnt/hadoop_tmp/mapred/staging/inosh/.staging is
>> not as expected. It is owned by inosh and permissions are rwxr-xr-x. The
>> directory must be owned by the submitter inosh or by inosh and permissions
>> must be rwx------)'
>> java.io.IOException: The ownership/permissions on the staging directory
>> hdfs://hadoop0.test.com:9000/mnt/hadoop_tmp/mapred/staging/inosh/.staging
>> is not as expected. It is owned by inosh and permissions are rwxr-xr-x. The
>> directory must be owned by the submitter inosh or by inosh and permissions
>> must be rwx------
>> at
>> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:108)
>> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:798)
>> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:792)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1123)
>> at
>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:792)
>> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:766)
>> at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:460)
>> at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136)
>> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:129)
>> at
>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:62)
>> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1351)
>> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1126)
>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:934)
>> at
>> org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:201)
>> at
>> org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:187)
>> ...
>>
>>
>> When we get this error we go to hadoop name node and execute following
>> command to set proper permission to the folder mentioned in the exception.
>> ./hadoop dfs -chmod 700 /mnt/hadoop_tmp/mapred/staging/inosh/.staging
>>
>>
>> We would like know why this happens and how to fix this permanently.
>>
>>
>> Thanks and Regards,
>> Inosh
>>
>
>
--
Harshit Mathur
Re: File permission of Hadoop staging directory getting changed time
to time
Posted by Harshit Mathur <ma...@gmail.com>.
try running with sudo hive.
On Mon, Apr 6, 2015 at 1:40 PM, Inosh Goonewardena <in...@gmail.com>
wrote:
> Hi All,
>
> Really appreciate any input on this issue.
>
> Thanks and Regards,
> Inosh
>
> On Tue, Jan 20, 2015 at 10:29 PM, Inosh Goonewardena <in...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> We have setup a Hadoop cluster using hadoop 1.0.4 and we use Hive to
>> submit map/reduce jobs. But we have noticed that these map/reduce job
>> submission failing time to time due to a permission issue occurring in
>> Hadoop. When this happens, we have to use the hadoop hdfs command and
>> change the file permission of mapreduce staging directory. Following is the
>> error that is getting logged.
>>
>> [2015-01-16 05:30:13,314] ERROR {org.apache.hadoop.hive.ql.exec.Task} -
>> Job Submission failed with exception 'java.io.IOException(The
>> ownership/permissions on the staging directory hdfs://
>> hadoop0.test.com:9000/mnt/hadoop_tmp/mapred/staging/inosh/.staging is
>> not as expected. It is owned by inosh and permissions are rwxr-xr-x. The
>> directory must be owned by the submitter inosh or by inosh and permissions
>> must be rwx------)'
>> java.io.IOException: The ownership/permissions on the staging directory
>> hdfs://hadoop0.test.com:9000/mnt/hadoop_tmp/mapred/staging/inosh/.staging
>> is not as expected. It is owned by inosh and permissions are rwxr-xr-x. The
>> directory must be owned by the submitter inosh or by inosh and permissions
>> must be rwx------
>> at
>> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:108)
>> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:798)
>> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:792)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1123)
>> at
>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:792)
>> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:766)
>> at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:460)
>> at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136)
>> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:129)
>> at
>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:62)
>> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1351)
>> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1126)
>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:934)
>> at
>> org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:201)
>> at
>> org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:187)
>> ...
>>
>>
>> When we get this error we go to hadoop name node and execute following
>> command to set proper permission to the folder mentioned in the exception.
>> ./hadoop dfs -chmod 700 /mnt/hadoop_tmp/mapred/staging/inosh/.staging
>>
>>
>> We would like know why this happens and how to fix this permanently.
>>
>>
>> Thanks and Regards,
>> Inosh
>>
>
>
--
Harshit Mathur
Re: File permission of Hadoop staging directory getting changed time
to time
Posted by Harshit Mathur <ma...@gmail.com>.
try running with sudo hive.
On Mon, Apr 6, 2015 at 1:40 PM, Inosh Goonewardena <in...@gmail.com>
wrote:
> Hi All,
>
> Really appreciate any input on this issue.
>
> Thanks and Regards,
> Inosh
>
> On Tue, Jan 20, 2015 at 10:29 PM, Inosh Goonewardena <in...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> We have setup a Hadoop cluster using hadoop 1.0.4 and we use Hive to
>> submit map/reduce jobs. But we have noticed that these map/reduce job
>> submission failing time to time due to a permission issue occurring in
>> Hadoop. When this happens, we have to use the hadoop hdfs command and
>> change the file permission of mapreduce staging directory. Following is the
>> error that is getting logged.
>>
>> [2015-01-16 05:30:13,314] ERROR {org.apache.hadoop.hive.ql.exec.Task} -
>> Job Submission failed with exception 'java.io.IOException(The
>> ownership/permissions on the staging directory hdfs://
>> hadoop0.test.com:9000/mnt/hadoop_tmp/mapred/staging/inosh/.staging is
>> not as expected. It is owned by inosh and permissions are rwxr-xr-x. The
>> directory must be owned by the submitter inosh or by inosh and permissions
>> must be rwx------)'
>> java.io.IOException: The ownership/permissions on the staging directory
>> hdfs://hadoop0.test.com:9000/mnt/hadoop_tmp/mapred/staging/inosh/.staging
>> is not as expected. It is owned by inosh and permissions are rwxr-xr-x. The
>> directory must be owned by the submitter inosh or by inosh and permissions
>> must be rwx------
>> at
>> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:108)
>> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:798)
>> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:792)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1123)
>> at
>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:792)
>> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:766)
>> at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:460)
>> at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136)
>> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:129)
>> at
>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:62)
>> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1351)
>> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1126)
>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:934)
>> at
>> org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:201)
>> at
>> org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:187)
>> ...
>>
>>
>> When we get this error we go to hadoop name node and execute following
>> command to set proper permission to the folder mentioned in the exception.
>> ./hadoop dfs -chmod 700 /mnt/hadoop_tmp/mapred/staging/inosh/.staging
>>
>>
>> We would like know why this happens and how to fix this permanently.
>>
>>
>> Thanks and Regards,
>> Inosh
>>
>
>
--
Harshit Mathur