You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hemanth Yamijala (JIRA)" <ji...@apache.org> on 2009/02/13 09:33:59 UTC

[jira] Created: (HADOOP-5248) Job directories could remain undeleted in some scenarios after job completes.

Job directories could remain undeleted in some scenarios after job completes.
-----------------------------------------------------------------------------

                 Key: HADOOP-5248
                 URL: https://issues.apache.org/jira/browse/HADOOP-5248
             Project: Hadoop Core
          Issue Type: Bug
          Components: mapred
            Reporter: Hemanth Yamijala


I observed a couple of times that when a job has completed, its job directories were not cleaned up. In discussion, it seems like there is a condition when only reduces from a job are run on a machine and no maps, the TT does not get a signal from the JT to delete the files and could be left behind. FYI, JVM reuse was enabled at the time. I can confirm that 'KillJobAction' was not received by the TT.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-5248) Job directories could remain undeleted in some scenarios after job completes.

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-5248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-5248:
-------------------------------------

         Priority: Blocker  (was: Major)
    Fix Version/s: 0.20.0

After discussion with Sameer and Devaraj, marking this a blocker for Hadoop 0.20.

> Job directories could remain undeleted in some scenarios after job completes.
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-5248
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5248
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Hemanth Yamijala
>            Priority: Blocker
>             Fix For: 0.20.0
>
>
> I observed a couple of times that when a job has completed, its job directories were not cleaned up. In discussion, it seems like there is a condition when only reduces from a job are run on a machine and no maps, the TT does not get a signal from the JT to delete the files and could be left behind. FYI, JVM reuse was enabled at the time. I can confirm that 'KillJobAction' was not received by the TT.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-5248) Job directories could remain undeleted in some scenarios after job completes.

Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-5248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-5248:
--------------------------------

    Attachment: 5248.patch

Attaching a patch. Testing in progress.

> Job directories could remain undeleted in some scenarios after job completes.
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-5248
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5248
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Hemanth Yamijala
>            Assignee: Devaraj Das
>            Priority: Blocker
>             Fix For: 0.20.0
>
>         Attachments: 5248.patch
>
>
> I observed a couple of times that when a job has completed, its job directories were not cleaned up. In discussion, it seems like there is a condition when only reduces from a job are run on a machine and no maps, the TT does not get a signal from the JT to delete the files and could be left behind. FYI, JVM reuse was enabled at the time. I can confirm that 'KillJobAction' was not received by the TT.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Resolved: (HADOOP-5248) Job directories could remain undeleted in some scenarios after job completes.

Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-5248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das resolved HADOOP-5248.
---------------------------------

    Resolution: Fixed

I just committed the testcase.

> Job directories could remain undeleted in some scenarios after job completes.
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-5248
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5248
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Hemanth Yamijala
>            Assignee: Devaraj Das
>            Priority: Blocker
>             Fix For: 0.20.0
>
>         Attachments: 5248-testcase.patch, 5248.patch
>
>
> I observed a couple of times that when a job has completed, its job directories were not cleaned up. In discussion, it seems like there is a condition when only reduces from a job are run on a machine and no maps, the TT does not get a signal from the JT to delete the files and could be left behind. FYI, JVM reuse was enabled at the time. I can confirm that 'KillJobAction' was not received by the TT.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5248) Job directories could remain undeleted in some scenarios after job completes.

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677027#action_12677027 ] 

Hudson commented on HADOOP-5248:
--------------------------------

Integrated in Hadoop-trunk #766 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/766/])
    . A testcase that checks for the existence of job directory after the job completes. Fails if it exists. Contributed by Devaraj Das.


> Job directories could remain undeleted in some scenarios after job completes.
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-5248
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5248
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Hemanth Yamijala
>            Assignee: Devaraj Das
>            Priority: Blocker
>             Fix For: 0.20.0
>
>         Attachments: 5248-testcase.patch, 5248.patch
>
>
> I observed a couple of times that when a job has completed, its job directories were not cleaned up. In discussion, it seems like there is a condition when only reduces from a job are run on a machine and no maps, the TT does not get a signal from the JT to delete the files and could be left behind. FYI, JVM reuse was enabled at the time. I can confirm that 'KillJobAction' was not received by the TT.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Assigned: (HADOOP-5248) Job directories could remain undeleted in some scenarios after job completes.

Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-5248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das reassigned HADOOP-5248:
-----------------------------------

    Assignee: Devaraj Das

> Job directories could remain undeleted in some scenarios after job completes.
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-5248
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5248
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Hemanth Yamijala
>            Assignee: Devaraj Das
>            Priority: Blocker
>             Fix For: 0.20.0
>
>
> I observed a couple of times that when a job has completed, its job directories were not cleaned up. In discussion, it seems like there is a condition when only reduces from a job are run on a machine and no maps, the TT does not get a signal from the JT to delete the files and could be left behind. FYI, JVM reuse was enabled at the time. I can confirm that 'KillJobAction' was not received by the TT.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-5248) Job directories could remain undeleted in some scenarios after job completes.

Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-5248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-5248:
--------------------------------

    Attachment: 5248-testcase.patch

The earlier patch I submitted can be ignored. HADOOP-5247 addresses the problem better by broadcasting a KillJobAction to all trackers upon job termination. The attached patch is a testcase that currently fails but passes with the patch for HADOOP-5247.

> Job directories could remain undeleted in some scenarios after job completes.
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-5248
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5248
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Hemanth Yamijala
>            Assignee: Devaraj Das
>            Priority: Blocker
>             Fix For: 0.20.0
>
>         Attachments: 5248-testcase.patch, 5248.patch
>
>
> I observed a couple of times that when a job has completed, its job directories were not cleaned up. In discussion, it seems like there is a condition when only reduces from a job are run on a machine and no maps, the TT does not get a signal from the JT to delete the files and could be left behind. FYI, JVM reuse was enabled at the time. I can confirm that 'KillJobAction' was not received by the TT.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.