You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by "Chandra Mohan, Ananda Vel Murugan" <An...@honeywell.com> on 2012/08/24 06:01:46 UTC
Running map reduce job from a web application
Hi,
I am developing a web application in Java which would be hosted in a machine outside of my Hadoop cluster. User account configuration would be different between the machines.
I am using the following code in my servlet to run the map-reduce job.
final Configuration config = new Configuration();
config.set("mapred.job.tracker", "<jobtracker.hostname>:9001");
config.set("fs.default.name", "hdfs://<namenode>:9000");
config.set("hadoop.tmp.dir", "/var/lib/hadoop/data-store/hadoop-hadoop");
config.set("hadoop.job.ugi", "hadoop,hadoop");
UserGroupInformation ugi1 = UserGroupInformation.createRemoteUser("hadoop");
Job job = new Job(config);
job.setJarByClass(ImportTsv.class);
job.setJobName("ImportTsv");
job.getConfiguration().set("user", "hadoop");
job.waitForCompletion(true);
job.submit();
Iterator iter = config.iterator();
while(iter.hasNext())
{
Object obj = iter.next();
System.out.println(obj);
}
It is not working and failing with following authentication error
Exception in thread "main" org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=E317376, access=WRITE, inode="staging":hadoop:supergroup:rwxr-xr-x
Appreciate your inputs for fixing this.
Regards,
Anand.C
Re: Running map reduce job from a web application
Posted by Bertrand Dechoux <de...@gmail.com>.
I haven't been confronted to your use case but your error message
explicitly says that you are not passing correctly the user "hadoop".
You should use the doAs() method on UserGroupInformation in order to
execute the job with this particular user.
http://mail-archives.apache.org/mod_mbox/hadoop-general/201009.mbox/%3CAANLkTiknJ_SzRux7KhjhxVfUU9FBkNgvYnkpbz3G_+a4@mail.gmail.com%3E
Regards
Bertrand
On Fri, Aug 24, 2012 at 6:01 AM, Chandra Mohan, Ananda Vel Murugan <
Ananda.Murugan@honeywell.com> wrote:
> Hi, ****
>
> ** **
>
> I am developing a web application in Java which would be hosted in a
> machine outside of my Hadoop cluster. User account configuration would be
> different between the machines. ****
>
> ** **
>
> I am using the following code in my servlet to run the map-reduce job.****
>
> ** **
>
> ** **
>
> *final* Configuration config = *new* Configuration();****
>
> config.set("mapred.job.tracker", "<jobtracker.hostname>:9001"
> );****
>
> config.set("fs.default.name", "hdfs://<namenode>:9000");****
>
> config.set("hadoop.tmp.dir",
> "/var/lib/hadoop/data-store/hadoop-hadoop");****
>
> config.set("hadoop.job.ugi", "hadoop,hadoop");****
>
> UserGroupInformation *ugi1* = UserGroupInformation.*
> createRemoteUser*("hadoop");****
>
> Job job = *new* Job(config);****
>
> job.setJarByClass(ImportTsv.*class*);****
>
> job.setJobName("ImportTsv");****
>
> job.getConfiguration().set("user", "hadoop");****
>
> job.waitForCompletion(*true*);****
>
> job.submit();****
>
> *Iterator* iter = config.iterator();****
>
> *while*(iter.hasNext())****
>
> {****
>
> Object obj = iter.next();****
>
> System.*out*.println(obj);****
>
> }****
>
> ** **
>
> It is not working and failing with following authentication error****
>
> ** **
>
> Exception in thread "main" *
> org.apache.hadoop.security.AccessControlException*: *
> org.apache.hadoop.security.AccessControlException*: Permission denied:
> user=E317376, access=WRITE, inode="staging":hadoop:supergroup:rwxr-xr-x***
> *
>
> ** **
>
> Appreciate your inputs for fixing this.****
>
> ** **
>
> Regards,****
>
> Anand.C ****
>
> ** **
>
> ** **
>
--
Bertrand Dechoux
Re: Running map reduce job from a web application
Posted by Bertrand Dechoux <de...@gmail.com>.
I haven't been confronted to your use case but your error message
explicitly says that you are not passing correctly the user "hadoop".
You should use the doAs() method on UserGroupInformation in order to
execute the job with this particular user.
http://mail-archives.apache.org/mod_mbox/hadoop-general/201009.mbox/%3CAANLkTiknJ_SzRux7KhjhxVfUU9FBkNgvYnkpbz3G_+a4@mail.gmail.com%3E
Regards
Bertrand
On Fri, Aug 24, 2012 at 6:01 AM, Chandra Mohan, Ananda Vel Murugan <
Ananda.Murugan@honeywell.com> wrote:
> Hi, ****
>
> ** **
>
> I am developing a web application in Java which would be hosted in a
> machine outside of my Hadoop cluster. User account configuration would be
> different between the machines. ****
>
> ** **
>
> I am using the following code in my servlet to run the map-reduce job.****
>
> ** **
>
> ** **
>
> *final* Configuration config = *new* Configuration();****
>
> config.set("mapred.job.tracker", "<jobtracker.hostname>:9001"
> );****
>
> config.set("fs.default.name", "hdfs://<namenode>:9000");****
>
> config.set("hadoop.tmp.dir",
> "/var/lib/hadoop/data-store/hadoop-hadoop");****
>
> config.set("hadoop.job.ugi", "hadoop,hadoop");****
>
> UserGroupInformation *ugi1* = UserGroupInformation.*
> createRemoteUser*("hadoop");****
>
> Job job = *new* Job(config);****
>
> job.setJarByClass(ImportTsv.*class*);****
>
> job.setJobName("ImportTsv");****
>
> job.getConfiguration().set("user", "hadoop");****
>
> job.waitForCompletion(*true*);****
>
> job.submit();****
>
> *Iterator* iter = config.iterator();****
>
> *while*(iter.hasNext())****
>
> {****
>
> Object obj = iter.next();****
>
> System.*out*.println(obj);****
>
> }****
>
> ** **
>
> It is not working and failing with following authentication error****
>
> ** **
>
> Exception in thread "main" *
> org.apache.hadoop.security.AccessControlException*: *
> org.apache.hadoop.security.AccessControlException*: Permission denied:
> user=E317376, access=WRITE, inode="staging":hadoop:supergroup:rwxr-xr-x***
> *
>
> ** **
>
> Appreciate your inputs for fixing this.****
>
> ** **
>
> Regards,****
>
> Anand.C ****
>
> ** **
>
> ** **
>
--
Bertrand Dechoux
Re: Running map reduce job from a web application
Posted by Bertrand Dechoux <de...@gmail.com>.
I haven't been confronted to your use case but your error message
explicitly says that you are not passing correctly the user "hadoop".
You should use the doAs() method on UserGroupInformation in order to
execute the job with this particular user.
http://mail-archives.apache.org/mod_mbox/hadoop-general/201009.mbox/%3CAANLkTiknJ_SzRux7KhjhxVfUU9FBkNgvYnkpbz3G_+a4@mail.gmail.com%3E
Regards
Bertrand
On Fri, Aug 24, 2012 at 6:01 AM, Chandra Mohan, Ananda Vel Murugan <
Ananda.Murugan@honeywell.com> wrote:
> Hi, ****
>
> ** **
>
> I am developing a web application in Java which would be hosted in a
> machine outside of my Hadoop cluster. User account configuration would be
> different between the machines. ****
>
> ** **
>
> I am using the following code in my servlet to run the map-reduce job.****
>
> ** **
>
> ** **
>
> *final* Configuration config = *new* Configuration();****
>
> config.set("mapred.job.tracker", "<jobtracker.hostname>:9001"
> );****
>
> config.set("fs.default.name", "hdfs://<namenode>:9000");****
>
> config.set("hadoop.tmp.dir",
> "/var/lib/hadoop/data-store/hadoop-hadoop");****
>
> config.set("hadoop.job.ugi", "hadoop,hadoop");****
>
> UserGroupInformation *ugi1* = UserGroupInformation.*
> createRemoteUser*("hadoop");****
>
> Job job = *new* Job(config);****
>
> job.setJarByClass(ImportTsv.*class*);****
>
> job.setJobName("ImportTsv");****
>
> job.getConfiguration().set("user", "hadoop");****
>
> job.waitForCompletion(*true*);****
>
> job.submit();****
>
> *Iterator* iter = config.iterator();****
>
> *while*(iter.hasNext())****
>
> {****
>
> Object obj = iter.next();****
>
> System.*out*.println(obj);****
>
> }****
>
> ** **
>
> It is not working and failing with following authentication error****
>
> ** **
>
> Exception in thread "main" *
> org.apache.hadoop.security.AccessControlException*: *
> org.apache.hadoop.security.AccessControlException*: Permission denied:
> user=E317376, access=WRITE, inode="staging":hadoop:supergroup:rwxr-xr-x***
> *
>
> ** **
>
> Appreciate your inputs for fixing this.****
>
> ** **
>
> Regards,****
>
> Anand.C ****
>
> ** **
>
> ** **
>
--
Bertrand Dechoux
Re: Running map reduce job from a web application
Posted by Bertrand Dechoux <de...@gmail.com>.
I haven't been confronted to your use case but your error message
explicitly says that you are not passing correctly the user "hadoop".
You should use the doAs() method on UserGroupInformation in order to
execute the job with this particular user.
http://mail-archives.apache.org/mod_mbox/hadoop-general/201009.mbox/%3CAANLkTiknJ_SzRux7KhjhxVfUU9FBkNgvYnkpbz3G_+a4@mail.gmail.com%3E
Regards
Bertrand
On Fri, Aug 24, 2012 at 6:01 AM, Chandra Mohan, Ananda Vel Murugan <
Ananda.Murugan@honeywell.com> wrote:
> Hi, ****
>
> ** **
>
> I am developing a web application in Java which would be hosted in a
> machine outside of my Hadoop cluster. User account configuration would be
> different between the machines. ****
>
> ** **
>
> I am using the following code in my servlet to run the map-reduce job.****
>
> ** **
>
> ** **
>
> *final* Configuration config = *new* Configuration();****
>
> config.set("mapred.job.tracker", "<jobtracker.hostname>:9001"
> );****
>
> config.set("fs.default.name", "hdfs://<namenode>:9000");****
>
> config.set("hadoop.tmp.dir",
> "/var/lib/hadoop/data-store/hadoop-hadoop");****
>
> config.set("hadoop.job.ugi", "hadoop,hadoop");****
>
> UserGroupInformation *ugi1* = UserGroupInformation.*
> createRemoteUser*("hadoop");****
>
> Job job = *new* Job(config);****
>
> job.setJarByClass(ImportTsv.*class*);****
>
> job.setJobName("ImportTsv");****
>
> job.getConfiguration().set("user", "hadoop");****
>
> job.waitForCompletion(*true*);****
>
> job.submit();****
>
> *Iterator* iter = config.iterator();****
>
> *while*(iter.hasNext())****
>
> {****
>
> Object obj = iter.next();****
>
> System.*out*.println(obj);****
>
> }****
>
> ** **
>
> It is not working and failing with following authentication error****
>
> ** **
>
> Exception in thread "main" *
> org.apache.hadoop.security.AccessControlException*: *
> org.apache.hadoop.security.AccessControlException*: Permission denied:
> user=E317376, access=WRITE, inode="staging":hadoop:supergroup:rwxr-xr-x***
> *
>
> ** **
>
> Appreciate your inputs for fixing this.****
>
> ** **
>
> Regards,****
>
> Anand.C ****
>
> ** **
>
> ** **
>
--
Bertrand Dechoux