You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Luca Telloli <te...@yahoo-inc.com> on 2008/02/26 18:06:07 UTC
[HOD] hdfs:///mapredsystem directory
Hello everyone,
I have a problem with directory hdfs:///mapredsystem and I'm not sure
if it's a bug or my fault.
Not sure if this influences what follows, but I have two users, one is
"hadoop" who has sudo privileges on all the nodes, the other one is
"luca", who has normal privileges.
I see that each submitted job creates a hdfs:///mapredsystem directory,
created by (I guess) the hodring process. Problem is that it's not
cleaned up at the end of the process; for instance a use case would be:
- user hadoop allocates a cluster, the ringmaster is svr3, so a
/mapredsystem/svr3 directory is created
- user hadoop deallocates the cluster, but that directory is not cleaned up
- user luca allocates a cluster, and the first node chosen as ringmaster
is svr3, so hodring tries to write hdfs:///mapredsystem but it fails
- allocation succeeds, but there's no hodring running; looking at
0-jobtracker/logdir/hadoop.log under the temporary directory I can read:
2008-02-26 17:28:42,567 WARN org.apache.hadoop.mapred.JobTracker: Error
starting tracker: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.fs.permission.AccessControlException: Permission
denied: user=luca, access=WRITE,
inode="mapredsystem":hadoop:supergroup:rwxr-xr-x
Am I doing anything wrong?
Cheers,
Luca
Re: [HOD] hdfs:///mapredsystem directory
Posted by Hemanth Yamijala <yh...@yahoo-inc.com>.
Mahadev Konar wrote:
> Hi Luca,
> Can you do a ls -l on /mapredsystem and send the output? According to
> permissions for mapreduce the system directories created by jobtracker
> should be world writable so permissions should have worked as it is for
> hod.
>
>
No, it doesn't appear to be working that way. The mapred system
directory specified by HOD for a jobtracker is /mapredsystem/<hostname>.
When the job tracker starts, it creates this directory with permissions
rwx-wx-wx. It does not clean it up when it stops. But when it is started
again on the same node, I think (may be wrong here) it tries to clean up
the directory (possibly to handle crashes etc ?). This seems equivalent
to issuing a "hadoop dfs -rmr /mapredsystem/<hostname>"which in turn rm
-r of unix. So, it basically has to read the directory to recursively
delete; and because there are no read permissions, it fails.
A simple experiment to simulate the same set of DFS operations using 2
different users, without using mapred, give the same result. Enabling
read permissions allows the delete to work.
There appear several ways we can fix this, and that discussion should go
on HADOOP-2899. Will post my comments there.
Thanks
hemanth
RE: [HOD] hdfs:///mapredsystem directory
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
Hi Luca,
Can you do a ls -l on /mapredsystem and send the output? According to
permissions for mapreduce the system directories created by jobtracker
should be world writable so permissions should have worked as it is for
hod.
Regards
Mahadev
> -----Original Message-----
> From: Luca Telloli [mailto:telloli@yahoo-inc.com]
> Sent: Tuesday, February 26, 2008 9:44 AM
> To: core-user@hadoop.apache.org
> Subject: Re: [HOD] hdfs:///mapredsystem directory
>
> Hi Mahadev,
> I'm not sure the workaround can solve the problem, because it
> appears
> that a subdirectory is created under that directory with the name of
the
> hodring host. So if the next allocation made by a different user
chooses
> the same host, the permission problem might show up again. Unless
> obviously each time the user who allocated resources deletes that
> directory during deallocation, which would be a solution for this
problem.
>
> I filed a bug: HADOOP-2899
>
> Cheers,
> Luca
>
> Mahadev Konar wrote:
> > Hi Luca,
> > This seems like a bug. The JobTracker process tries to create this
> > directory if this does nto exist. And if you have two different
users
> > running hod clusters they will both try to create this directory and
> > since only one succeeds, with permissions the directory is owned by
the
> > user who created the cluster first.
> >
> > A work around to this solution is to create
> >
> > Hdfs:///mapredsystem manually and make it world writable.
> >
> > Please open a bug regarding this issue.
> >
> > Regards
> > Mahadev
> >
> >> -----Original Message-----
> >> From: Luca Telloli [mailto:telloli@yahoo-inc.com]
> >> Sent: Tuesday, February 26, 2008 9:06 AM
> >> To: core-user@hadoop.apache.org
> >> Subject: [HOD] hdfs:///mapredsystem directory
> >>
> >> Hello everyone,
> >> I have a problem with directory hdfs:///mapredsystem and I'm not
> >> sure
> >> if it's a bug or my fault.
> >>
> >> Not sure if this influences what follows, but I have two users, one
is
> >> "hadoop" who has sudo privileges on all the nodes, the other one is
> >> "luca", who has normal privileges.
> >>
> >> I see that each submitted job creates a hdfs:///mapredsystem
> > directory,
> >> created by (I guess) the hodring process. Problem is that it's not
> >> cleaned up at the end of the process; for instance a use case would
> > be:
> >> - user hadoop allocates a cluster, the ringmaster is svr3, so a
> >> /mapredsystem/svr3 directory is created
> >>
> >> - user hadoop deallocates the cluster, but that directory is not
> > cleaned
> >> up
> >>
> >> - user luca allocates a cluster, and the first node chosen as
> > ringmaster
> >> is svr3, so hodring tries to write hdfs:///mapredsystem but it
fails
> >>
> >> - allocation succeeds, but there's no hodring running; looking at
> >> 0-jobtracker/logdir/hadoop.log under the temporary directory I can
> > read:
> >> 2008-02-26 17:28:42,567 WARN org.apache.hadoop.mapred.JobTracker:
> > Error
> >> starting tracker: org.apache.hadoop.ipc.RemoteException:
> >> org.apache.hadoop.fs.permission.AccessControlException: Permission
> >> denied: user=luca, access=WRITE,
> >> inode="mapredsystem":hadoop:supergroup:rwxr-xr-x
> >>
> >> Am I doing anything wrong?
> >> Cheers,
> >> Luca
> >
Re: [HOD] hdfs:///mapredsystem directory
Posted by Luca Telloli <te...@yahoo-inc.com>.
Hi Mahadev,
I'm not sure the workaround can solve the problem, because it appears
that a subdirectory is created under that directory with the name of the
hodring host. So if the next allocation made by a different user chooses
the same host, the permission problem might show up again. Unless
obviously each time the user who allocated resources deletes that
directory during deallocation, which would be a solution for this problem.
I filed a bug: HADOOP-2899
Cheers,
Luca
Mahadev Konar wrote:
> Hi Luca,
> This seems like a bug. The JobTracker process tries to create this
> directory if this does nto exist. And if you have two different users
> running hod clusters they will both try to create this directory and
> since only one succeeds, with permissions the directory is owned by the
> user who created the cluster first.
>
> A work around to this solution is to create
>
> Hdfs:///mapredsystem manually and make it world writable.
>
> Please open a bug regarding this issue.
>
> Regards
> Mahadev
>
>> -----Original Message-----
>> From: Luca Telloli [mailto:telloli@yahoo-inc.com]
>> Sent: Tuesday, February 26, 2008 9:06 AM
>> To: core-user@hadoop.apache.org
>> Subject: [HOD] hdfs:///mapredsystem directory
>>
>> Hello everyone,
>> I have a problem with directory hdfs:///mapredsystem and I'm not
>> sure
>> if it's a bug or my fault.
>>
>> Not sure if this influences what follows, but I have two users, one is
>> "hadoop" who has sudo privileges on all the nodes, the other one is
>> "luca", who has normal privileges.
>>
>> I see that each submitted job creates a hdfs:///mapredsystem
> directory,
>> created by (I guess) the hodring process. Problem is that it's not
>> cleaned up at the end of the process; for instance a use case would
> be:
>> - user hadoop allocates a cluster, the ringmaster is svr3, so a
>> /mapredsystem/svr3 directory is created
>>
>> - user hadoop deallocates the cluster, but that directory is not
> cleaned
>> up
>>
>> - user luca allocates a cluster, and the first node chosen as
> ringmaster
>> is svr3, so hodring tries to write hdfs:///mapredsystem but it fails
>>
>> - allocation succeeds, but there's no hodring running; looking at
>> 0-jobtracker/logdir/hadoop.log under the temporary directory I can
> read:
>> 2008-02-26 17:28:42,567 WARN org.apache.hadoop.mapred.JobTracker:
> Error
>> starting tracker: org.apache.hadoop.ipc.RemoteException:
>> org.apache.hadoop.fs.permission.AccessControlException: Permission
>> denied: user=luca, access=WRITE,
>> inode="mapredsystem":hadoop:supergroup:rwxr-xr-x
>>
>> Am I doing anything wrong?
>> Cheers,
>> Luca
>
RE: [HOD] hdfs:///mapredsystem directory
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
Hi Luca,
This seems like a bug. The JobTracker process tries to create this
directory if this does nto exist. And if you have two different users
running hod clusters they will both try to create this directory and
since only one succeeds, with permissions the directory is owned by the
user who created the cluster first.
A work around to this solution is to create
Hdfs:///mapredsystem manually and make it world writable.
Please open a bug regarding this issue.
Regards
Mahadev
> -----Original Message-----
> From: Luca Telloli [mailto:telloli@yahoo-inc.com]
> Sent: Tuesday, February 26, 2008 9:06 AM
> To: core-user@hadoop.apache.org
> Subject: [HOD] hdfs:///mapredsystem directory
>
> Hello everyone,
> I have a problem with directory hdfs:///mapredsystem and I'm not
> sure
> if it's a bug or my fault.
>
> Not sure if this influences what follows, but I have two users, one is
> "hadoop" who has sudo privileges on all the nodes, the other one is
> "luca", who has normal privileges.
>
> I see that each submitted job creates a hdfs:///mapredsystem
directory,
> created by (I guess) the hodring process. Problem is that it's not
> cleaned up at the end of the process; for instance a use case would
be:
>
> - user hadoop allocates a cluster, the ringmaster is svr3, so a
> /mapredsystem/svr3 directory is created
>
> - user hadoop deallocates the cluster, but that directory is not
cleaned
> up
>
> - user luca allocates a cluster, and the first node chosen as
ringmaster
> is svr3, so hodring tries to write hdfs:///mapredsystem but it fails
>
> - allocation succeeds, but there's no hodring running; looking at
> 0-jobtracker/logdir/hadoop.log under the temporary directory I can
read:
>
> 2008-02-26 17:28:42,567 WARN org.apache.hadoop.mapred.JobTracker:
Error
> starting tracker: org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.fs.permission.AccessControlException: Permission
> denied: user=luca, access=WRITE,
> inode="mapredsystem":hadoop:supergroup:rwxr-xr-x
>
> Am I doing anything wrong?
> Cheers,
> Luca