You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2011/01/28 22:04:44 UTC
[jira] Created: (MAPREDUCE-2289) Permissions race can make
getStagingDir fail on local filesystem
Permissions race can make getStagingDir fail on local filesystem
----------------------------------------------------------------
Key: MAPREDUCE-2289
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2289
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: job submission
Affects Versions: 0.22.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Fix For: 0.22.0
I've observed the following race condition in TestFairSchedulerSystem which uses a MiniMRCluster on top of RawLocalFileSystem:
- two threads call getStagingDir at the same time
- Thread A checks fs.exists(stagingArea) and sees false
-- Calls mkdirs(stagingArea, JOB_DIR_PERMISSIONS)
--- mkdirs calls the Java mkdir API which makes the file with umask-based permissions
- Thread B runs, checks fs.exists(stagingArea) and sees true
-- checks permissions, sees the default permissions, and throws IOE
- Thread A resumes and sets correct permissions
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2289) Permissions race can make
getStagingDir fail on local filesystem
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2289:
-----------------------------------
Status: Patch Available (was: Open)
> Permissions race can make getStagingDir fail on local filesystem
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-2289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2289
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.22.0
>
> Attachments: mapreduce-2289.txt
>
>
> I've observed the following race condition in TestFairSchedulerSystem which uses a MiniMRCluster on top of RawLocalFileSystem:
> - two threads call getStagingDir at the same time
> - Thread A checks fs.exists(stagingArea) and sees false
> -- Calls mkdirs(stagingArea, JOB_DIR_PERMISSIONS)
> --- mkdirs calls the Java mkdir API which makes the file with umask-based permissions
> - Thread B runs, checks fs.exists(stagingArea) and sees true
> -- checks permissions, sees the default permissions, and throws IOE
> - Thread A resumes and sets correct permissions
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2289) Permissions race can make
getStagingDir fail on local filesystem
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13400947#comment-13400947 ]
Hudson commented on MAPREDUCE-2289:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #2386 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2386/])
MAPREDUCE-2289. Permissions race can make getStagingDir fail on local filesystem (ahmed via tucu) (Revision 1353750)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353750
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java
> Permissions race can make getStagingDir fail on local filesystem
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-2289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2289
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 1.1.0, 2.0.1-alpha
>
> Attachments: MAPREDUCE-2289_branch-1.0.patch, MAPREDUCE-2289_trunk.patch, mapreduce-2289.txt
>
>
> I've observed the following race condition in TestFairSchedulerSystem which uses a MiniMRCluster on top of RawLocalFileSystem:
> - two threads call getStagingDir at the same time
> - Thread A checks fs.exists(stagingArea) and sees false
> -- Calls mkdirs(stagingArea, JOB_DIR_PERMISSIONS)
> --- mkdirs calls the Java mkdir API which makes the file with umask-based permissions
> - Thread B runs, checks fs.exists(stagingArea) and sees true
> -- checks permissions, sees the default permissions, and throws IOE
> - Thread A resumes and sets correct permissions
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2289) Permissions race can make
getStagingDir fail on local filesystem
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2289:
-----------------------------------
Attachment: mapreduce-2289.txt
Here's a patch which implements the proposed change (fixing submission dir permissions if they're incorrect)
> Permissions race can make getStagingDir fail on local filesystem
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-2289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2289
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.22.0
>
> Attachments: mapreduce-2289.txt
>
>
> I've observed the following race condition in TestFairSchedulerSystem which uses a MiniMRCluster on top of RawLocalFileSystem:
> - two threads call getStagingDir at the same time
> - Thread A checks fs.exists(stagingArea) and sees false
> -- Calls mkdirs(stagingArea, JOB_DIR_PERMISSIONS)
> --- mkdirs calls the Java mkdir API which makes the file with umask-based permissions
> - Thread B runs, checks fs.exists(stagingArea) and sees true
> -- checks permissions, sees the default permissions, and throws IOE
> - Thread A resumes and sets correct permissions
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2289) Permissions race can make
getStagingDir fail on local filesystem
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293029#comment-13293029 ]
Hadoop QA commented on MAPREDUCE-2289:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12531712/MAPREDUCE-2289_trunk.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2449//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2449//console
This message is automatically generated.
> Permissions race can make getStagingDir fail on local filesystem
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-2289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2289
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.22.1
>
> Attachments: MAPREDUCE-2289_branch-1.0.patch, MAPREDUCE-2289_trunk.patch, mapreduce-2289.txt
>
>
> I've observed the following race condition in TestFairSchedulerSystem which uses a MiniMRCluster on top of RawLocalFileSystem:
> - two threads call getStagingDir at the same time
> - Thread A checks fs.exists(stagingArea) and sees false
> -- Calls mkdirs(stagingArea, JOB_DIR_PERMISSIONS)
> --- mkdirs calls the Java mkdir API which makes the file with umask-based permissions
> - Thread B runs, checks fs.exists(stagingArea) and sees true
> -- checks permissions, sees the default permissions, and throws IOE
> - Thread A resumes and sets correct permissions
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2289) Permissions race can make
getStagingDir fail on local filesystem
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2289:
-------------------------------------
Status: Open (was: Patch Available)
Sorry to come in late, the patch has gone stale. Can you please rebase? Thanks.
> Permissions race can make getStagingDir fail on local filesystem
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-2289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2289
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.22.0
>
> Attachments: mapreduce-2289.txt
>
>
> I've observed the following race condition in TestFairSchedulerSystem which uses a MiniMRCluster on top of RawLocalFileSystem:
> - two threads call getStagingDir at the same time
> - Thread A checks fs.exists(stagingArea) and sees false
> -- Calls mkdirs(stagingArea, JOB_DIR_PERMISSIONS)
> --- mkdirs calls the Java mkdir API which makes the file with umask-based permissions
> - Thread B runs, checks fs.exists(stagingArea) and sees true
> -- checks permissions, sees the default permissions, and throws IOE
> - Thread A resumes and sets correct permissions
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2289) Permissions race can make
getStagingDir fail on local filesystem
Posted by "Ahmed Radwan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan reassigned MAPREDUCE-2289:
---------------------------------------
Assignee: Ahmed Radwan (was: Todd Lipcon)
> Permissions race can make getStagingDir fail on local filesystem
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-2289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2289
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.22.1
>
> Attachments: MAPREDUCE-2289_branch-1.0.patch, MAPREDUCE-2289_trunk.patch, mapreduce-2289.txt
>
>
> I've observed the following race condition in TestFairSchedulerSystem which uses a MiniMRCluster on top of RawLocalFileSystem:
> - two threads call getStagingDir at the same time
> - Thread A checks fs.exists(stagingArea) and sees false
> -- Calls mkdirs(stagingArea, JOB_DIR_PERMISSIONS)
> --- mkdirs calls the Java mkdir API which makes the file with umask-based permissions
> - Thread B runs, checks fs.exists(stagingArea) and sees true
> -- checks permissions, sees the default permissions, and throws IOE
> - Thread A resumes and sets correct permissions
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2289) Permissions race can make
getStagingDir fail on local filesystem
Posted by "Ahmed Radwan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-2289:
------------------------------------
Status: Patch Available (was: Open)
> Permissions race can make getStagingDir fail on local filesystem
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-2289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2289
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.22.1
>
> Attachments: MAPREDUCE-2289_branch-1.0.patch, MAPREDUCE-2289_trunk.patch, mapreduce-2289.txt
>
>
> I've observed the following race condition in TestFairSchedulerSystem which uses a MiniMRCluster on top of RawLocalFileSystem:
> - two threads call getStagingDir at the same time
> - Thread A checks fs.exists(stagingArea) and sees false
> -- Calls mkdirs(stagingArea, JOB_DIR_PERMISSIONS)
> --- mkdirs calls the Java mkdir API which makes the file with umask-based permissions
> - Thread B runs, checks fs.exists(stagingArea) and sees true
> -- checks permissions, sees the default permissions, and throws IOE
> - Thread A resumes and sets correct permissions
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2289) Permissions race can make
getStagingDir fail on local filesystem
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated MAPREDUCE-2289:
------------------------------------------
Resolution: Fixed
Fix Version/s: (was: 0.22.1)
2.0.1-alpha
1.1.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Ahmed. Committed to trunk, branch-1 & branch-2.
> Permissions race can make getStagingDir fail on local filesystem
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-2289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2289
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 1.1.0, 2.0.1-alpha
>
> Attachments: MAPREDUCE-2289_branch-1.0.patch, MAPREDUCE-2289_trunk.patch, mapreduce-2289.txt
>
>
> I've observed the following race condition in TestFairSchedulerSystem which uses a MiniMRCluster on top of RawLocalFileSystem:
> - two threads call getStagingDir at the same time
> - Thread A checks fs.exists(stagingArea) and sees false
> -- Calls mkdirs(stagingArea, JOB_DIR_PERMISSIONS)
> --- mkdirs calls the Java mkdir API which makes the file with umask-based permissions
> - Thread B runs, checks fs.exists(stagingArea) and sees true
> -- checks permissions, sees the default permissions, and throws IOE
> - Thread A resumes and sets correct permissions
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2289) Permissions race can make
getStagingDir fail on local filesystem
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401418#comment-13401418 ]
Hudson commented on MAPREDUCE-2289:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #1121 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1121/])
MAPREDUCE-2289. Permissions race can make getStagingDir fail on local filesystem (ahmed via tucu) (Revision 1353750)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353750
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java
> Permissions race can make getStagingDir fail on local filesystem
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-2289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2289
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 1.1.0, 2.0.1-alpha
>
> Attachments: MAPREDUCE-2289_branch-1.0.patch, MAPREDUCE-2289_trunk.patch, mapreduce-2289.txt
>
>
> I've observed the following race condition in TestFairSchedulerSystem which uses a MiniMRCluster on top of RawLocalFileSystem:
> - two threads call getStagingDir at the same time
> - Thread A checks fs.exists(stagingArea) and sees false
> -- Calls mkdirs(stagingArea, JOB_DIR_PERMISSIONS)
> --- mkdirs calls the Java mkdir API which makes the file with umask-based permissions
> - Thread B runs, checks fs.exists(stagingArea) and sees true
> -- checks permissions, sees the default permissions, and throws IOE
> - Thread A resumes and sets correct permissions
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2289) Permissions race can make
getStagingDir fail on local filesystem
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988233#action_12988233 ]
Todd Lipcon commented on MAPREDUCE-2289:
----------------------------------------
The central issue here is that the Java mkdir API doesn't take a mode, so whenever we create a directory, it starts out with umask-based permissions until we can call chmod on it.
Seems to me it would be safe and fix the race if we had it *fix* permissions if they were wrong, rather than bailing out. Then the race would only cause a redundant chmod. Thoughts?
> Permissions race can make getStagingDir fail on local filesystem
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-2289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2289
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.22.0
>
>
> I've observed the following race condition in TestFairSchedulerSystem which uses a MiniMRCluster on top of RawLocalFileSystem:
> - two threads call getStagingDir at the same time
> - Thread A checks fs.exists(stagingArea) and sees false
> -- Calls mkdirs(stagingArea, JOB_DIR_PERMISSIONS)
> --- mkdirs calls the Java mkdir API which makes the file with umask-based permissions
> - Thread B runs, checks fs.exists(stagingArea) and sees true
> -- checks permissions, sees the default permissions, and throws IOE
> - Thread A resumes and sets correct permissions
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Closed] (MAPREDUCE-2289) Permissions race can make
getStagingDir fail on local filesystem
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy closed MAPREDUCE-2289.
------------------------------------
> Permissions race can make getStagingDir fail on local filesystem
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-2289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2289
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.23.3, 2.0.2-alpha
>
> Attachments: MAPREDUCE-2289_branch-1.0.patch, MAPREDUCE-2289_trunk.patch, mapreduce-2289.txt
>
>
> I've observed the following race condition in TestFairSchedulerSystem which uses a MiniMRCluster on top of RawLocalFileSystem:
> - two threads call getStagingDir at the same time
> - Thread A checks fs.exists(stagingArea) and sees false
> -- Calls mkdirs(stagingArea, JOB_DIR_PERMISSIONS)
> --- mkdirs calls the Java mkdir API which makes the file with umask-based permissions
> - Thread B runs, checks fs.exists(stagingArea) and sees true
> -- checks permissions, sees the default permissions, and throws IOE
> - Thread A resumes and sets correct permissions
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2289) Permissions race can make
getStagingDir fail on local filesystem
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13400972#comment-13400972 ]
Hudson commented on MAPREDUCE-2289:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2456 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2456/])
MAPREDUCE-2289. Permissions race can make getStagingDir fail on local filesystem (ahmed via tucu) (Revision 1353750)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353750
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java
> Permissions race can make getStagingDir fail on local filesystem
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-2289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2289
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 1.1.0, 2.0.1-alpha
>
> Attachments: MAPREDUCE-2289_branch-1.0.patch, MAPREDUCE-2289_trunk.patch, mapreduce-2289.txt
>
>
> I've observed the following race condition in TestFairSchedulerSystem which uses a MiniMRCluster on top of RawLocalFileSystem:
> - two threads call getStagingDir at the same time
> - Thread A checks fs.exists(stagingArea) and sees false
> -- Calls mkdirs(stagingArea, JOB_DIR_PERMISSIONS)
> --- mkdirs calls the Java mkdir API which makes the file with umask-based permissions
> - Thread B runs, checks fs.exists(stagingArea) and sees true
> -- checks permissions, sees the default permissions, and throws IOE
> - Thread A resumes and sets correct permissions
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2289) Permissions race can make
getStagingDir fail on local filesystem
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408371#comment-13408371 ]
Hudson commented on MAPREDUCE-2289:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #305 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/305/])
svn merge -c 1353750 FIXES: MAPREDUCE-2289. Permissions race can make getStagingDir fail on local filesystem (ahmed via tucu) (Revision 1358400)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358400
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java
> Permissions race can make getStagingDir fail on local filesystem
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-2289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2289
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 1.1.0, 0.23.3, 2.0.1-alpha
>
> Attachments: MAPREDUCE-2289_branch-1.0.patch, MAPREDUCE-2289_trunk.patch, mapreduce-2289.txt
>
>
> I've observed the following race condition in TestFairSchedulerSystem which uses a MiniMRCluster on top of RawLocalFileSystem:
> - two threads call getStagingDir at the same time
> - Thread A checks fs.exists(stagingArea) and sees false
> -- Calls mkdirs(stagingArea, JOB_DIR_PERMISSIONS)
> --- mkdirs calls the Java mkdir API which makes the file with umask-based permissions
> - Thread B runs, checks fs.exists(stagingArea) and sees true
> -- checks permissions, sees the default permissions, and throws IOE
> - Thread A resumes and sets correct permissions
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2289) Permissions race can make
getStagingDir fail on local filesystem
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999793#comment-12999793 ]
Hadoop QA commented on MAPREDUCE-2289:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12469984/mapreduce-2289.txt
against trunk revision 1074251.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/34//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/34//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/34//console
This message is automatically generated.
> Permissions race can make getStagingDir fail on local filesystem
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-2289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2289
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.22.0
>
> Attachments: mapreduce-2289.txt
>
>
> I've observed the following race condition in TestFairSchedulerSystem which uses a MiniMRCluster on top of RawLocalFileSystem:
> - two threads call getStagingDir at the same time
> - Thread A checks fs.exists(stagingArea) and sees false
> -- Calls mkdirs(stagingArea, JOB_DIR_PERMISSIONS)
> --- mkdirs calls the Java mkdir API which makes the file with umask-based permissions
> - Thread B runs, checks fs.exists(stagingArea) and sees true
> -- checks permissions, sees the default permissions, and throws IOE
> - Thread A resumes and sets correct permissions
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2289) Permissions race can make
getStagingDir fail on local filesystem
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401318#comment-13401318 ]
Hudson commented on MAPREDUCE-2289:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #1088 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1088/])
MAPREDUCE-2289. Permissions race can make getStagingDir fail on local filesystem (ahmed via tucu) (Revision 1353750)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353750
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java
> Permissions race can make getStagingDir fail on local filesystem
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-2289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2289
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 1.1.0, 2.0.1-alpha
>
> Attachments: MAPREDUCE-2289_branch-1.0.patch, MAPREDUCE-2289_trunk.patch, mapreduce-2289.txt
>
>
> I've observed the following race condition in TestFairSchedulerSystem which uses a MiniMRCluster on top of RawLocalFileSystem:
> - two threads call getStagingDir at the same time
> - Thread A checks fs.exists(stagingArea) and sees false
> -- Calls mkdirs(stagingArea, JOB_DIR_PERMISSIONS)
> --- mkdirs calls the Java mkdir API which makes the file with umask-based permissions
> - Thread B runs, checks fs.exists(stagingArea) and sees true
> -- checks permissions, sees the default permissions, and throws IOE
> - Thread A resumes and sets correct permissions
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2289) Permissions race can make
getStagingDir fail on local filesystem
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13400926#comment-13400926 ]
Alejandro Abdelnur commented on MAPREDUCE-2289:
-----------------------------------------------
+1
> Permissions race can make getStagingDir fail on local filesystem
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-2289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2289
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.22.1
>
> Attachments: MAPREDUCE-2289_branch-1.0.patch, MAPREDUCE-2289_trunk.patch, mapreduce-2289.txt
>
>
> I've observed the following race condition in TestFairSchedulerSystem which uses a MiniMRCluster on top of RawLocalFileSystem:
> - two threads call getStagingDir at the same time
> - Thread A checks fs.exists(stagingArea) and sees false
> -- Calls mkdirs(stagingArea, JOB_DIR_PERMISSIONS)
> --- mkdirs calls the Java mkdir API which makes the file with umask-based permissions
> - Thread B runs, checks fs.exists(stagingArea) and sees true
> -- checks permissions, sees the default permissions, and throws IOE
> - Thread A resumes and sets correct permissions
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2289) Permissions race can make
getStagingDir fail on local filesystem
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-2289:
-------------------------------------------
Fix Version/s: 0.23.3
> Permissions race can make getStagingDir fail on local filesystem
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-2289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2289
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 1.1.0, 0.23.3, 2.0.1-alpha
>
> Attachments: MAPREDUCE-2289_branch-1.0.patch, MAPREDUCE-2289_trunk.patch, mapreduce-2289.txt
>
>
> I've observed the following race condition in TestFairSchedulerSystem which uses a MiniMRCluster on top of RawLocalFileSystem:
> - two threads call getStagingDir at the same time
> - Thread A checks fs.exists(stagingArea) and sees false
> -- Calls mkdirs(stagingArea, JOB_DIR_PERMISSIONS)
> --- mkdirs calls the Java mkdir API which makes the file with umask-based permissions
> - Thread B runs, checks fs.exists(stagingArea) and sees true
> -- checks permissions, sees the default permissions, and throws IOE
> - Thread A resumes and sets correct permissions
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2289) Permissions race can make
getStagingDir fail on local filesystem
Posted by "Ahmed Radwan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-2289:
------------------------------------
Attachment: MAPREDUCE-2289_trunk.patch
MAPREDUCE-2289_branch-1.0.patch
Here are updated patched that apply cleanly to both trunk and branch-1.0.
> Permissions race can make getStagingDir fail on local filesystem
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-2289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2289
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.22.1
>
> Attachments: MAPREDUCE-2289_branch-1.0.patch, MAPREDUCE-2289_trunk.patch, mapreduce-2289.txt
>
>
> I've observed the following race condition in TestFairSchedulerSystem which uses a MiniMRCluster on top of RawLocalFileSystem:
> - two threads call getStagingDir at the same time
> - Thread A checks fs.exists(stagingArea) and sees false
> -- Calls mkdirs(stagingArea, JOB_DIR_PERMISSIONS)
> --- mkdirs calls the Java mkdir API which makes the file with umask-based permissions
> - Thread B runs, checks fs.exists(stagingArea) and sees true
> -- checks permissions, sees the default permissions, and throws IOE
> - Thread A resumes and sets correct permissions
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2289) Permissions race can make
getStagingDir fail on local filesystem
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401001#comment-13401001 ]
Hudson commented on MAPREDUCE-2289:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2404 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2404/])
MAPREDUCE-2289. Permissions race can make getStagingDir fail on local filesystem (ahmed via tucu) (Revision 1353750)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353750
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java
> Permissions race can make getStagingDir fail on local filesystem
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-2289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2289
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 1.1.0, 2.0.1-alpha
>
> Attachments: MAPREDUCE-2289_branch-1.0.patch, MAPREDUCE-2289_trunk.patch, mapreduce-2289.txt
>
>
> I've observed the following race condition in TestFairSchedulerSystem which uses a MiniMRCluster on top of RawLocalFileSystem:
> - two threads call getStagingDir at the same time
> - Thread A checks fs.exists(stagingArea) and sees false
> -- Calls mkdirs(stagingArea, JOB_DIR_PERMISSIONS)
> --- mkdirs calls the Java mkdir API which makes the file with umask-based permissions
> - Thread B runs, checks fs.exists(stagingArea) and sees true
> -- checks permissions, sees the default permissions, and throws IOE
> - Thread A resumes and sets correct permissions
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira