You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by YongChul Kwon <go...@gmail.com> on 2008/06/26 21:10:21 UTC
Sharing Hadoop cluster among multiple users
Hello all,
I deployed hadoop to a small cluster. The HDFS is running as user A.
Now user B comes in and wants to run a simple Map-Reduce task. The
Map-Reduce client creates all shared files in correct location
'/tmp/hadoop-B/mapred/system/...'. However, JobTracker, running as
user A, always tries to read the job related files from
'/tmp/hadoop-A/mapred/system/...'.
The questions are following.
a) Does everybody who wants to run map-reduce job have to launch one's
own JobTracker?
b) Do I have to set HOD for this? I really don't want to do this extra
since B is currently the only user running Map-Reduce.
Thank you,
YongChul
Re: Sharing Hadoop cluster among multiple users
Posted by YongChul Kwon <go...@gmail.com>.
Hi,
I meant conflicts in file system name space not the permission.
Anyway, I found a related JIRA entry for this refers the setting of
mapred.system.dir.
Thank you for your responses! :-)
YongChul
On Thu, Jun 26, 2008 at 3:40 PM, Haijun Cao <ha...@kindsight.net> wrote:
> I don't think having the username in the path provides safety either
> (other than it forces separate cluster).
>
> Hadoop file system has access control (but without authentication) that
> prevents users stepping on each other's toes.
>
>
>
> Haijun
>
> -----Original Message-----
> From: YongChul Kwon [mailto:godslord@gmail.com]
> Sent: Thursday, June 26, 2008 12:45 PM
> To: core-user@hadoop.apache.org
> Subject: Re: Sharing Hadoop cluster among multiple users
>
> Hi Hiajun,
>
> Yes, they, path related to shared things for map reduce, set to
> default. I thought there is a reason for the default values containing
> 'username' in them. Is it safe to use some global path, say dropping
> the username reference from the default values?
>
> Thank you,
> YongChul
>
> On Thu, Jun 26, 2008 at 12:19 PM, Haijun Cao <ha...@kindsight.net>
> wrote:
>> Is it because you leave the mapred.system.dir as default (see
>> hadoop-default.xml)?
>>
>> Haijun
>>
>> -----Original Message-----
>> From: YongChul Kwon [mailto:godslord@gmail.com]
>> Sent: Thursday, June 26, 2008 12:10 PM
>> To: core-user@hadoop.apache.org
>> Subject: Sharing Hadoop cluster among multiple users
>>
>> Hello all,
>>
>> I deployed hadoop to a small cluster. The HDFS is running as user A.
>> Now user B comes in and wants to run a simple Map-Reduce task. The
>> Map-Reduce client creates all shared files in correct location
>> '/tmp/hadoop-B/mapred/system/...'. However, JobTracker, running as
>> user A, always tries to read the job related files from
>> '/tmp/hadoop-A/mapred/system/...'.
>>
>> The questions are following.
>> a) Does everybody who wants to run map-reduce job have to launch one's
>> own JobTracker?
>> b) Do I have to set HOD for this? I really don't want to do this extra
>> since B is currently the only user running Map-Reduce.
>>
>> Thank you,
>> YongChul
>>
>
>
>
> --
> YongChul
>
--
YongChul
RE: Sharing Hadoop cluster among multiple users
Posted by Haijun Cao <ha...@kindsight.net>.
I don't think having the username in the path provides safety either
(other than it forces separate cluster).
Hadoop file system has access control (but without authentication) that
prevents users stepping on each other's toes.
Haijun
-----Original Message-----
From: YongChul Kwon [mailto:godslord@gmail.com]
Sent: Thursday, June 26, 2008 12:45 PM
To: core-user@hadoop.apache.org
Subject: Re: Sharing Hadoop cluster among multiple users
Hi Hiajun,
Yes, they, path related to shared things for map reduce, set to
default. I thought there is a reason for the default values containing
'username' in them. Is it safe to use some global path, say dropping
the username reference from the default values?
Thank you,
YongChul
On Thu, Jun 26, 2008 at 12:19 PM, Haijun Cao <ha...@kindsight.net>
wrote:
> Is it because you leave the mapred.system.dir as default (see
> hadoop-default.xml)?
>
> Haijun
>
> -----Original Message-----
> From: YongChul Kwon [mailto:godslord@gmail.com]
> Sent: Thursday, June 26, 2008 12:10 PM
> To: core-user@hadoop.apache.org
> Subject: Sharing Hadoop cluster among multiple users
>
> Hello all,
>
> I deployed hadoop to a small cluster. The HDFS is running as user A.
> Now user B comes in and wants to run a simple Map-Reduce task. The
> Map-Reduce client creates all shared files in correct location
> '/tmp/hadoop-B/mapred/system/...'. However, JobTracker, running as
> user A, always tries to read the job related files from
> '/tmp/hadoop-A/mapred/system/...'.
>
> The questions are following.
> a) Does everybody who wants to run map-reduce job have to launch one's
> own JobTracker?
> b) Do I have to set HOD for this? I really don't want to do this extra
> since B is currently the only user running Map-Reduce.
>
> Thank you,
> YongChul
>
--
YongChul
Re: Sharing Hadoop cluster among multiple users
Posted by YongChul Kwon <go...@gmail.com>.
Hi Hiajun,
Yes, they, path related to shared things for map reduce, set to
default. I thought there is a reason for the default values containing
'username' in them. Is it safe to use some global path, say dropping
the username reference from the default values?
Thank you,
YongChul
On Thu, Jun 26, 2008 at 12:19 PM, Haijun Cao <ha...@kindsight.net> wrote:
> Is it because you leave the mapred.system.dir as default (see
> hadoop-default.xml)?
>
> Haijun
>
> -----Original Message-----
> From: YongChul Kwon [mailto:godslord@gmail.com]
> Sent: Thursday, June 26, 2008 12:10 PM
> To: core-user@hadoop.apache.org
> Subject: Sharing Hadoop cluster among multiple users
>
> Hello all,
>
> I deployed hadoop to a small cluster. The HDFS is running as user A.
> Now user B comes in and wants to run a simple Map-Reduce task. The
> Map-Reduce client creates all shared files in correct location
> '/tmp/hadoop-B/mapred/system/...'. However, JobTracker, running as
> user A, always tries to read the job related files from
> '/tmp/hadoop-A/mapred/system/...'.
>
> The questions are following.
> a) Does everybody who wants to run map-reduce job have to launch one's
> own JobTracker?
> b) Do I have to set HOD for this? I really don't want to do this extra
> since B is currently the only user running Map-Reduce.
>
> Thank you,
> YongChul
>
--
YongChul
RE: Sharing Hadoop cluster among multiple users
Posted by Haijun Cao <ha...@kindsight.net>.
Is it because you leave the mapred.system.dir as default (see
hadoop-default.xml)?
Haijun
-----Original Message-----
From: YongChul Kwon [mailto:godslord@gmail.com]
Sent: Thursday, June 26, 2008 12:10 PM
To: core-user@hadoop.apache.org
Subject: Sharing Hadoop cluster among multiple users
Hello all,
I deployed hadoop to a small cluster. The HDFS is running as user A.
Now user B comes in and wants to run a simple Map-Reduce task. The
Map-Reduce client creates all shared files in correct location
'/tmp/hadoop-B/mapred/system/...'. However, JobTracker, running as
user A, always tries to read the job related files from
'/tmp/hadoop-A/mapred/system/...'.
The questions are following.
a) Does everybody who wants to run map-reduce job have to launch one's
own JobTracker?
b) Do I have to set HOD for this? I really don't want to do this extra
since B is currently the only user running Map-Reduce.
Thank you,
YongChul