You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Eli Collins (JIRA)" <ji...@apache.org> on 2011/01/03 23:58:46 UTC
[jira] Created: (MAPREDUCE-2238) Undeletable build directories
Undeletable build directories
------------------------------
Key: MAPREDUCE-2238
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build, test
Affects Versions: 0.23.0
Reporter: Eli Collins
The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
Building remotely on hadoop7
hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
at hudson.FilePath.act(FilePath.java:749)
at hudson.FilePath.act(FilePath.java:735)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
at hudson.model.Run.run(Run.java:1324)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:139)
Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977492#action_12977492 ]
Nigel Daley commented on MAPREDUCE-2238:
----------------------------------------
Hudson slave runs as the user 'hudson'. Here is the directory listing that is failing. Why would the permissions on the dir be changed to 311?
{code}
root@h7:/grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001# ls -al
total 36
drwx------ 9 hudson hudson 4096 2010-12-30 13:14 .
drwxr-xr-x 3 hudson hudson 4096 2010-12-30 13:14 ..
d-wx--x--x 2 hudson hudson 4096 2010-12-30 13:11 attempt_20101230131139886_0001_m_000000_0
drwx------ 2 hudson hudson 4096 2010-12-30 13:12 attempt_20101230131139886_0001_m_000000_1
drwx------ 2 hudson hudson 4096 2010-12-30 13:11 attempt_20101230131139886_0001_m_000001_0
drwx------ 2 hudson hudson 4096 2010-12-30 13:12 attempt_20101230131139886_0001_m_000003_1
drwx------ 2 hudson hudson 4096 2010-12-30 13:11 attempt_20101230131139886_0001_m_000005_0
drwx------ 2 hudson hudson 4096 2010-12-30 13:12 attempt_20101230131139886_0001_m_000006_0
drwx------ 2 hudson hudson 4096 2010-12-30 13:11 attempt_20101230131139886_0001_m_000007_0
root@h7:/grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001# ls -al attempt_20101230131139886_0001_m_000000_0
total 16
d-wx--x--x 2 hudson hudson 4096 2010-12-30 13:11 .
drwx------ 9 hudson hudson 4096 2010-12-30 13:14 ..
-rw-r--r-- 1 hudson hudson 205 2010-12-30 13:11 log.index
-rw-r--r-- 1 hudson hudson 0 2010-12-30 13:11 stderr
-rw-r--r-- 1 hudson hudson 1687 2010-12-30 13:11 stdout
{code}
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.23.0
> Reporter: Eli Collins
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993934#comment-12993934 ]
Hudson commented on MAPREDUCE-2238:
-----------------------------------
Integrated in Hadoop-Mapreduce-22-branch #33 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/33/])
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.22.0
> Reporter: Eli Collins
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.22.0
>
> Attachments: mapreduce-2238.txt, mapreduce-2238.txt, mapreduce-2238.txt
>
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2238) Undeletable build directories
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2238:
-----------------------------------
Attachment: mapreduce-2238.txt
I don't know that this is the issue, but the new setPermissions code is definitely prone to races. If two threads tried to setPermissions on the same directory at once, it could definitely end up with an incorrect result.
This patch makes setPermissions threadsafe at least against other invocations of the same method. Worth a shot to apply this and see if the problems go away?
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.23.0
> Reporter: Eli Collins
> Attachments: mapreduce-2238.txt
>
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978222#action_12978222 ]
Todd Lipcon commented on MAPREDUCE-2238:
----------------------------------------
Linked MAPREDUCE-1972 which has some logs which might be related
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.23.0
> Reporter: Eli Collins
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12983812#action_12983812 ]
Todd Lipcon commented on MAPREDUCE-2238:
----------------------------------------
Unit tests pass except known failures
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.22.0
> Reporter: Eli Collins
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.22.0
>
> Attachments: mapreduce-2238.txt, mapreduce-2238.txt, mapreduce-2238.txt
>
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2238) Undeletable build directories
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2238:
-----------------------------------
Attachment: mapreduce-2238.txt
Here's a patch which gets rid of the racy PermissionsHandler code and replaces it with calls to LocalFileSystem.setPermission. When combined with HADOOP-7110 this will actually be *more* efficient and also avoid the bug described in this JIRA.
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.22.0
> Reporter: Eli Collins
> Assignee: Todd Lipcon
> Fix For: 0.22.0
>
> Attachments: mapreduce-2238.txt, mapreduce-2238.txt
>
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12981520#action_12981520 ]
Todd Lipcon commented on MAPREDUCE-2238:
----------------------------------------
Saw this again on a build here. This time the undeletable userlog directory was created by TestMiniMRWithDFSWithDistinctUsers.
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.23.0
> Reporter: Eli Collins
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12982755#action_12982755 ]
Todd Lipcon commented on MAPREDUCE-2238:
----------------------------------------
Bummer, I had this patch in a branch on our internal Hudson and the problem happened anyway... so it doesn't look like it's an issue with thread safety on setPermissions
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.23.0
> Reporter: Eli Collins
> Attachments: mapreduce-2238.txt
>
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12982913#action_12982913 ]
Todd Lipcon commented on MAPREDUCE-2238:
----------------------------------------
Another clue on this one:
root@ubuntu64-build01:/home/hudson/production/workspace/CDH3-todd-security/build/test# stat logs/userlogs/job_20110116195813351_0001/attempt_20110116195813351_0001_m_000001_0/
File: `logs/userlogs/job_20110116195813351_0001/attempt_20110116195813351_0001_m_000001_0/'
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 801h/2049d Inode: 4376820 Links: 2
Access: (0311/d-wx--x--x) Uid: ( 1065/ hudson) Gid: ( 1065/ hudson)
Access: 2011-01-17 15:10:22.000000000 -0800
Modify: 2011-01-16 19:58:49.000000000 -0800
Change: 2011-01-16 19:58:49.000000000 -0800
Note the change time. Now looking at what tests ran during that change time:
root@ubuntu64-build01:/home/hudson/production/workspace/CDH3-todd-security/build/test# grep -l '2011-01-16 20:58:4' *
TEST-org.apache.hadoop.mapred.TestTaskLogsTruncater.xml
Seems like TestTaskLogsTruncater is probably the culprit. Also suspicious is that that same test failed a test run a few builds prior to this.
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.23.0
> Reporter: Eli Collins
> Attachments: mapreduce-2238.txt
>
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
Posted by "Greg Roelofs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978551#action_12978551 ]
Greg Roelofs commented on MAPREDUCE-2238:
-----------------------------------------
bq. I guess it could be a test timing out right as a setPermissions is done, interrupting in the middle... but seems pretty unlikely, don't you think?
Yes. I'm guessing it's more subtle than that and lies within the core MR code or the JVM. The fact that I see it semi-frequently on NFS (that is, more frequent than Hudson or production) suggests either timing (NFS is slow), perhaps via an erroneous assumption of synchronous behavior, or else an erroneous assumption of an infallible system call. It could be other things as well, of course, but those seem to me like the most probable candidates.
bq. I agree we could work around it for the tests, but I'm nervous whether we will see this issue crop up in production. Have you guys at Yahoo seen this on any clusters running secure YDH?
To clarify, I was suggesting working around it in the MR code itself, not realizing that the Hudson backtrace wasn't using MR code at all. (Well, apparently.) So I'm not sure where that leaves us, other than trying to fix the actual set-permissions problem. Seems like no one's basic deleteRecursive() implementation includes an option to attempt a chmod() before failing on bad permissions?
Anyway, yes, I _think_ we've seen it in production with 0.20S or later, but it wasn't while I was on call, so I might be remembering a different issue with similar symptoms. Sorry...there are lots of interesting failure modes in Hadoop, and my memory is finite. :-)
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.23.0
> Reporter: Eli Collins
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
Posted by "Greg Roelofs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978593#action_12978593 ]
Greg Roelofs commented on MAPREDUCE-2238:
-----------------------------------------
bq. But that's why Hadoop's so much fun to work on! If it just worked all the time we'd be bored.
We are entirely in agreement. :-)
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.23.0
> Reporter: Eli Collins
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978209#action_12978209 ]
Todd Lipcon commented on MAPREDUCE-2238:
----------------------------------------
I guess it could be a test timing out right as a setPermissions is done, interrupting in the middle... but seems pretty unlikely, don't you think?
I agree we could work around it for the tests, but I'm nervous whether we will see this issue crop up in production. Have you guys at Yahoo seen this on any clusters running secure YDH?
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.23.0
> Reporter: Eli Collins
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
Posted by "Greg Roelofs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978109#action_12978109 ]
Greg Roelofs commented on MAPREDUCE-2238:
-----------------------------------------
Race condition/lack of locking?
FWIW, I've seen this exact same bug several times when running "ant test" on an NFS tree. (IIRC, it screws up all subsequent tests, too, until the bogus directory is manually removed or chmod'd.)
If the bug proves too elusive--and it's definitely intermittent and not at all frequent--a hackish workaround would be just to catch the IOException, attempt a chmod 700, and reattempt the deletion.
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.23.0
> Reporter: Eli Collins
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978561#action_12978561 ]
Todd Lipcon commented on MAPREDUCE-2238:
----------------------------------------
I think we should do the following:
- change out the implementation of setPermissions to not use the Java APIs, but rather the old style system("chmod") approach
- implement a proper chmod call in libhadoop (JNI) to avoid the fork for production systems where the fork is too expensive
bq. Sorry...there are lots of interesting failure modes in Hadoop, and my memory is finite
:) But that's why Hadoop's so much fun to work on! If it just worked all the time we'd be bored.
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.23.0
> Reporter: Eli Collins
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12983267#action_12983267 ]
Todd Lipcon commented on MAPREDUCE-2238:
----------------------------------------
Spent some time adding logging and looping the tests to figure out this problem. I think I have it cracked.
The issue is not multiple threads calling setPermission() on the same process, but rather a case where one thread is calling setPermission on the *parent* directory of a file where another thread (actually another entire process) is calling setPermission.
In particular, these two invocations race:
2011-01-18 09:00:40,958 INFO tasktracker.Localizer (Localizer.java:setPermissions(129)) - Thread[TaskLauncher for MAP tasks,5,main]: About to set permissions on /data/1/todd/cdh/repos/cdh3/hadoop-0.20/build/test/logs/userlogs/job_20110118090037816_0001
java.lang.Exception
at org.apache.hadoop.mapreduce.server.tasktracker.Localizer$PermissionsHandler.setPermissions(Localizer.java:129)
at org.apache.hadoop.mapreduce.server.tasktracker.Localizer.initializeJobLogDir(Localizer.java:429)
at org.apache.hadoop.mapred.TaskTracker.initializeJobLogDir(TaskTracker.java:1072)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:969)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2209)
2011-01-18 09:00:40,985 INFO tasktracker.Localizer (Localizer.java:setPermissions(129)) - Thread[Thread-213,5,main]: About to set permissions on /data/1/todd/cdh/repos/cdh3/hadoop-0.20/build/test/logs/userlogs/job_20110118090037816_0001/attempt_20110118090037816_0001_m_000005_0
java.lang.Exception
at org.apache.hadoop.mapreduce.server.tasktracker.Localizer$PermissionsHandler.setPermissions(Localizer.java:129)
at org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:285)
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
The above traces are from an 0.20 branch but I imagine it's the same deal on trunk.
The issue is that the top invocation flips the job_<id> directory to 000 momentarily. During that time, the stat/chmod calls for the attempt directory fail with EACCES, which can leave the attempt directory with the wrong permissions. I have strace output which shows this as well.
I think we should do away with this Java API nonsense altogether, link in a normal chmod call, and use fork by default when native isn't available.
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.23.0
> Reporter: Eli Collins
> Attachments: mapreduce-2238.txt
>
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12983421#action_12983421 ]
Eli Collins commented on MAPREDUCE-2238:
----------------------------------------
+1 latest patch looks good.
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.22.0
> Reporter: Eli Collins
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.22.0
>
> Attachments: mapreduce-2238.txt, mapreduce-2238.txt
>
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (MAPREDUCE-2238) Undeletable build directories
Posted by "Kihwal Lee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050073#comment-13050073 ]
Kihwal Lee commented on MAPREDUCE-2238:
---------------------------------------
We've seen this in 0.20-security builds. May be we should put HADOOP-7110 and this to 0.20-security. I see they are in CDH3.
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.22.0
> Reporter: Eli Collins
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.22.0
>
> Attachments: mapreduce-2238.txt, mapreduce-2238.txt, mapreduce-2238.txt
>
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986169#action_12986169 ]
Hudson commented on MAPREDUCE-2238:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #583 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/583/])
MAPREDUCE-2238. Fix permissions handling to avoid leaving undeletable directories in local dirs. Contributed by Todd Lipcon
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.22.0
> Reporter: Eli Collins
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.22.0
>
> Attachments: mapreduce-2238.txt, mapreduce-2238.txt, mapreduce-2238.txt
>
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977503#action_12977503 ]
Todd Lipcon commented on MAPREDUCE-2238:
----------------------------------------
My hunch is that this is caused by some kind of bug in the new Localizer.PermissionsHandler.setPermissions originally introduced by MAPREDUCE-842.
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.23.0
> Reporter: Eli Collins
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2238) Undeletable build directories
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2238:
-----------------------------------
Priority: Critical (was: Major)
Affects Version/s: (was: 0.23.0)
0.22.0
Fix Version/s: 0.22.0
Changing to critical for 0.22, since the bug is now understood, and it ends up killing Hudson on a regular basis
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.22.0
> Reporter: Eli Collins
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.22.0
>
> Attachments: mapreduce-2238.txt, mapreduce-2238.txt
>
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-2238) Undeletable build directories
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon reassigned MAPREDUCE-2238:
--------------------------------------
Assignee: Todd Lipcon
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.23.0
> Reporter: Eli Collins
> Assignee: Todd Lipcon
> Attachments: mapreduce-2238.txt
>
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2238) Undeletable build directories
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2238:
-----------------------------------
Status: Patch Available (was: Open)
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.22.0
> Reporter: Eli Collins
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.22.0
>
> Attachments: mapreduce-2238.txt, mapreduce-2238.txt, mapreduce-2238.txt
>
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2238) Undeletable build directories
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2238:
-----------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed to trunk and branch, thanks for review Eli, and thanks to Greg for helping brainstorm.
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.22.0
> Reporter: Eli Collins
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.22.0
>
> Attachments: mapreduce-2238.txt, mapreduce-2238.txt, mapreduce-2238.txt
>
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12982915#action_12982915 ]
Todd Lipcon commented on MAPREDUCE-2238:
----------------------------------------
Oops scratch that, it's 19:58, not 20:58... so not TestTaskLogsTruncater.
Interestingly, the tests running around that area are:
TestJobStatusPersistency: Last log message at 19:58:09
TestJobTrackerInstrumentation: First log msg at 19:59:03
So we have a gap between those two.
According to the test log, this test ran in between them:
[junit] Running org.apache.hadoop.mapred.TestJobSysDirWithDFS
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 50.818 sec
So I think *that* was the test that made the undeletable dir.
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.23.0
> Reporter: Eli Collins
> Attachments: mapreduce-2238.txt
>
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2238) Undeletable build directories
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2238:
-----------------------------------
Attachment: mapreduce-2238.txt
Previous patch had a slight change where the Job ACL file was 600 instead of 700. Not clear why it should be 700 (it's not executable!) but it shouldn't be fixed as part of this JIRA (caused some localization tests to fail).
Will resubmit this new patch for tests and open a new JIRA to fix permissions there to make more sense.
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.22.0
> Reporter: Eli Collins
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.22.0
>
> Attachments: mapreduce-2238.txt, mapreduce-2238.txt, mapreduce-2238.txt
>
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (MAPREDUCE-2238) Undeletable build directories
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016946#comment-13016946 ]
Hudson commented on MAPREDUCE-2238:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #643 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/])
> Undeletable build directories
> ------------------------------
>
> Key: MAPREDUCE-2238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
> Affects Versions: 0.22.0
> Reporter: Eli Collins
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.22.0
>
> Attachments: mapreduce-2238.txt, mapreduce-2238.txt, mapreduce-2238.txt
>
>
> The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
> Building remotely on hadoop7
> hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7
> at hudson.FilePath.act(FilePath.java:749)
> at hudson.FilePath.act(FilePath.java:735)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
> at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
> at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
> at hudson.model.Run.run(Run.java:1324)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
> at hudson.model.ResourceController.execute(ResourceController.java:88)
> at hudson.model.Executor.run(Executor.java:139)
> Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_000000_0
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira