You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Jay Vyas <ja...@gmail.com> on 2013/04/12 13:11:37 UTC

JobSubmissionFiles: past , present, and future?

Hi guys: 

I'm curious about the changes and future of the JobSubmissionFiles class.

Grepping around on the web I'm finding some code snippets that suggest that hadoop security is not handled the same way on the staging directory as before:

http://javasourcecode.org/html/open-source/hadoop/hadoop-0.20.203.0/org/apache/hadoop/mapreduce/JobSubmissionFiles.java.html

http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201210.mbox/%3CCAOcnVr0EYLSckXaOCPNm7kBZWpHvCdjBBX5A+AzeS_S6PwsbkQ@mail.gmail.com%3E

But I'm having trouble definitively pinning this to versions.

Why the difference in the if/else logic here and what is the future
Of permissions on .staging?

Re: JobSubmissionFiles: past , present, and future?

Posted by Jay Vyas <ja...@gmail.com>.
To update on this, it was just pointed out to me by matt farrallee 
that the auto fix of permissions is for a failsafe 
in case of a race condition, and not meant to mend bad permissions in all cases:

https://github.com/apache/hadoop-common/commit/f25dc04795a0e9836e3f237c802bfc1fe8a243ad

Something to keep in mind - if you see the "fixing staging permissions" error message alot
Then there might be a more systemic problem in your fs... At least, that was the case for us.

On Apr 12, 2013, at 6:11 AM, Jay Vyas <ja...@gmail.com> wrote:

> Hi guys: 
> 
> I'm curious about the changes and future of the JobSubmissionFiles class.
> 
> Grepping around on the web I'm finding some code snippets that suggest that hadoop security is not handled the same way on the staging directory as before:
> 
> http://javasourcecode.org/html/open-source/hadoop/hadoop-0.20.203.0/org/apache/hadoop/mapreduce/JobSubmissionFiles.java.html
> 
> http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201210.mbox/%3CCAOcnVr0EYLSckXaOCPNm7kBZWpHvCdjBBX5A+AzeS_S6PwsbkQ@mail.gmail.com%3E
> 
> But I'm having trouble definitively pinning this to versions.
> 
> Why the difference in the if/else logic here and what is the future
> Of permissions on .staging?