You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Alejandro Abdelnur (JIRA)" <ji...@apache.org> on 2012/04/27 20:14:55 UTC

[jira] [Created] (HADOOP-8325) remove FileSystem JVM shutdown hook that does closeAll()

Alejandro Abdelnur created HADOOP-8325:
------------------------------------------

             Summary: remove FileSystem JVM shutdown hook that does closeAll()
                 Key: HADOOP-8325
                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 2.0.0
            Reporter: Alejandro Abdelnur
            Assignee: Alejandro Abdelnur
            Priority: Critical
             Fix For: 2.0.0


FileSystem adds a JVM shutdown hook when a filesystem instance is cached.

MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.

This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-8325:
---------------------------------------

    Attachment: HADOOP-8325.patch
    
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263975#comment-13263975 ] 

Hadoop QA commented on HADOOP-8325:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12524914/HADOOP-8325.patch
  against trunk revision .

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 2 new or modified test files.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 eclipse:eclipse.  The patch built with eclipse:eclipse.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these unit tests:
                  org.apache.hadoop.io.TestSequenceFile
                  org.apache.hadoop.fs.viewfs.TestViewFsTrash

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/895//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/895//console

This message is automatically generated.
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13407056#comment-13407056 ] 

Hudson commented on HADOOP-8325:
--------------------------------

Integrated in Hadoop-Hdfs-0.23-Build #304 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/304/])
    svn merge -c 1333224 FIXES: HADOOP-8342. HDFS command fails with exception following merge of HADOOP-8325 (tucu) (Revision 1356970)
svn merge -c 1332345 FIXES: HADOOP-8325. Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook (tucu) (Revision 1356966)

     Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356970
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java

bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356966
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileContext.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/RunJar.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileContextDeleteOnExit.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestShutdownHookManager.java

                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0-alpha
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 0.23.3
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-8325:
---------------------------------------

    Status: Patch Available  (was: Open)
    
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-8325:
---------------------------------------

    Status: Patch Available  (was: Open)
    
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265141#comment-13265141 ] 

Hudson commented on HADOOP-8325:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #2155 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2155/])
    HADOOP-8325. Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook (tucu) (Revision 1332345)

     Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1332345
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileContext.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/RunJar.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileContextDeleteOnExit.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestShutdownHookManager.java

                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264202#comment-13264202 ] 

Alejandro Abdelnur commented on HADOOP-8325:
--------------------------------------------

My bad, missed TestFileContextDeleteOnExit failure (somehow I've seen only the first 2). Going for Nicholas' second suggestion. 
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264085#comment-13264085 ] 

Alejandro Abdelnur commented on HADOOP-8325:
--------------------------------------------

test failures seem unrelated
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-8325:
---------------------------------------

    Attachment: HADOOP-8325.patch
    
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267410#comment-13267410 ] 

Hudson commented on HADOOP-8325:
--------------------------------

Integrated in Hadoop-Hdfs-trunk #1033 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1033/])
    HADOOP-8342. HDFS command fails with exception following merge of HADOOP-8325 (tucu) (Revision 1333224)

     Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333224
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java

                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Ravi Prakash (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264354#comment-13264354 ] 

Ravi Prakash commented on HADOOP-8325:
--------------------------------------

Hey Tucu! What functionality does this Shutdown Hook Manager provide that the JVM shutdown hook manager does not? http://docs.oracle.com/javase/1.4.2/docs/guide/lang/hook-design.html states
bq. The thread can be created in the proper thread group, given the correct priority, context, and privileges, and so forth. 
We can set priorities of the existing shutdown thread and since the JVM uses a preemptive, priority based scheduling algorithm, they will in essence run in order of priorities (two threads may have the same priorities but that is besides the point).
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Ravi Prakash (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264618#comment-13264618 ] 

Ravi Prakash commented on HADOOP-8325:
--------------------------------------

Thanks Tucu! You are right. I had not thought of that.

I'm on the fence because we are serializing a process which should happen fast (so parallelizing is good). Like you already mentioned 
bq. 1* Add a method in the FileSystem to disable its shutdown hook. MRAppMaster would disable it and call closeAll() explicitly (it already does).
This is already present (by setting conf.setBoolean("fs.automatic.close", false) ) So I'm not sure what problem this JIRA is fixing. Or are we planning for the future? I'm fine with the latter answer being yes. I'm just curious.

The patch looks good for what it intends to do, although the latest one doesn't include the fixes you made in response to Nicholas' comments.
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-8325:
---------------------------------------

    Status: Open  (was: Patch Available)
    
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264056#comment-13264056 ] 

Hadoop QA commented on HADOOP-8325:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12524937/HADOOP-8325.patch
  against trunk revision .

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 2 new or modified test files.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 eclipse:eclipse.  The patch built with eclipse:eclipse.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these unit tests:
                  org.apache.hadoop.fs.viewfs.TestViewFsTrash
                  org.apache.hadoop.io.TestSequenceFile
                  org.apache.hadoop.fs.TestFileContextDeleteOnExit

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/898//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/898//console

This message is automatically generated.
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265187#comment-13265187 ] 

Hudson commented on HADOOP-8325:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #2172 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2172/])
    HADOOP-8325. Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook (tucu) (Revision 1332345)

     Result = ABORTED
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1332345
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileContext.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/RunJar.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileContextDeleteOnExit.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestShutdownHookManager.java

                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264335#comment-13264335 ] 

Hadoop QA commented on HADOOP-8325:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12524980/HADOOP-8325.patch
  against trunk revision .

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 2 new or modified test files.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 eclipse:eclipse.  The patch built with eclipse:eclipse.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these unit tests:
                  org.apache.hadoop.io.TestSequenceFile

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/903//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/903//console

This message is automatically generated.
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) remove FileSystem JVM shutdown hook that does closeAll()

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263864#comment-13263864 ] 

Alejandro Abdelnur commented on HADOOP-8325:
--------------------------------------------

I can think of 2 options:

1* Add a method in the FileSystem to disable its shutdown hook. MRAppMaster would disable it and call closeAll() explicitly (it already does).

2* Remove the FileSystem shutdown hook completely. For process that don't call closeAll() the NN eventually will close down the orphan sockets.
                
> remove FileSystem JVM shutdown hook that does closeAll()
> --------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265819#comment-13265819 ] 

Hudson commented on HADOOP-8325:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk #1066 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1066/])
    HADOOP-8325. Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook (tucu) (Revision 1332345)

     Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1332345
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileContext.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/RunJar.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileContextDeleteOnExit.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestShutdownHookManager.java

                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265785#comment-13265785 ] 

Hudson commented on HADOOP-8325:
--------------------------------

Integrated in Hadoop-Hdfs-trunk #1031 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1031/])
    HADOOP-8325. Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook (tucu) (Revision 1332345)

     Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1332345
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileContext.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/RunJar.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileContextDeleteOnExit.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestShutdownHookManager.java

                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-8325:
---------------------------------------

    Attachment: HADOOP-8325.patch
    
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-8325:
---------------------------------------

    Attachment: HADOOP-8325.patch

patch addressing Nicholas' #1 comment.
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) remove FileSystem JVM shutdown hook that does closeAll()

Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263883#comment-13263883 ] 

Todd Lipcon commented on HADOOP-8325:
-------------------------------------

Here's another option:

3* Add a shutdown hook "manager" to Hadoop common, which installs a singular shutdown hook. Other components can register their own hooks with the manager, with a given priority. The hooks would then get run in priority order.
                
> remove FileSystem JVM shutdown hook that does closeAll()
> --------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263989#comment-13263989 ] 

Alejandro Abdelnur commented on HADOOP-8325:
--------------------------------------------

test failures seem unrelated
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265126#comment-13265126 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-8325:
------------------------------------------------

+1 the new patch looks good.
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-8325:
---------------------------------------

    Status: Patch Available  (was: Open)
    
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264125#comment-13264125 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-8325:
------------------------------------------------

After changed to TreeSet, hasShutdownHook and removeShutdownHook does not work unless the priority is also passed.  The failure of TestFileContextDeleteOnExit is related to this.  The problem is that HookEntry.equals(..) and HookEntry.compareTo(..) are not consistent, i.e. HookEntry.compareTo(that) == 0 does not imply HookEntry.equals == true.  Two possible solutions:
- check priority in equals and pass priority in hasShutdownHook and removeShutdownHook.
- use HashSet and remove compareTo(..) from HookEntry.  Use comparator in sorting.  The comparator passed should sort reverse order.  Since it is a separated comparator, I think it won't be counter intuitive.
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-8325:
---------------------------------------

      Resolution: Fixed
    Hadoop Flags: Reviewed
          Status: Resolved  (was: Patch Available)

committed to trunk and branch-2
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-8325:
---------------------------------------

    Status: Open  (was: Patch Available)

need to retroffiting all shutdownhooks in common
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266951#comment-13266951 ] 

Hudson commented on HADOOP-8325:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #2174 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2174/])
    HADOOP-8342. HDFS command fails with exception following merge of HADOOP-8325 (tucu) (Revision 1333224)

     Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333224
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java

                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266946#comment-13266946 ] 

Hudson commented on HADOOP-8325:
--------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #2248 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2248/])
    HADOOP-8342. HDFS command fails with exception following merge of HADOOP-8325 (tucu) (Revision 1333224)

     Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333224
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java

                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264396#comment-13264396 ] 

Alejandro Abdelnur commented on HADOOP-8325:
--------------------------------------------

@Ravi, thread priority/group/context/privileges does not ensure order/serialization of execution given a number of processors/core greater than one (and I assume it is not guaranteed even in the case of a single core).
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265107#comment-13265107 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-8325:
------------------------------------------------

Hi Alejandro,

hooks is a synchronized set but we still have to manually synchronize it when iterating it.  So, we need to synchronize the first line of getShutdownHooksInOrder() which copies the hooks to a new list.
{code}
+  List<Runnable> getShutdownHooksInOrder() {
+    List<HookEntry> list = new ArrayList<HookEntry>(MGR.hooks);
{code}

BTW, what should be expected for adding/removing hooks during shutdown?  I think we probably should throw an IllegalStateException.
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Joseph Evans updated HADOOP-8325:
----------------------------------------

    Fix Version/s:     (was: 2.0.0-alpha)
                   0.23.3
    
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0-alpha
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 0.23.3
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264033#comment-13264033 ] 

Alejandro Abdelnur commented on HADOOP-8325:
--------------------------------------------

Thanks Nicholas,

#1, will do
#2, i thought about this, but it seemed counter intuitive to reverse the comparison. As this is done only once and the number of hooks would always be small, I don't see it as an issue. Because of that I'd prefer to keep it like it is.
#3, that woud require the iterator() method to be public giving access to anybody to the shutdown hooks and potentially invoking them outside of the ShutdownHookManager. The getShutdownHooksInOrder() method is package private only for test purposes (otherwise it would be private). Regarding the copying, same rationale as in #2.
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-8325:
---------------------------------------

    Summary: Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook  (was: remove FileSystem JVM shutdown hook that does closeAll())

IMO, Option #3 seems the best choice (updating summary to reflect that)
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264021#comment-13264021 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-8325:
------------------------------------------------

Minor comments:
- use TreeSet instead of HashSet, then it will be sorted.
- change HookEntry.compareTo(..) to return o.priority - priority; then we don't have to reverse the order.
- change ShutdownHookManager to implement Iterable instead of copying the hooks to a list in getShutdownHooksInOrder().

                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265140#comment-13265140 ] 

Hudson commented on HADOOP-8325:
--------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #2229 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2229/])
    HADOOP-8325. Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook (tucu) (Revision 1332345)

     Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1332345
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileContext.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/RunJar.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileContextDeleteOnExit.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestShutdownHookManager.java

                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264664#comment-13264664 ] 

Alejandro Abdelnur commented on HADOOP-8325:
--------------------------------------------

@Ravi,

the latest patch integrates Nicholas comments, reverse comparator and HashSet.

the 'fs.automatic.close' is per filesystem instance. It is not that we don't want to close the filesystem, but we want to close it after we finished using it. The patch fixes the problem mentioned in the description of the JIRA. Furthermore, there are other shutdown hooks in  FileContext and in MR that may require the filesystems to be still open.
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267444#comment-13267444 ] 

Hudson commented on HADOOP-8325:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk #1068 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1068/])
    HADOOP-8342. HDFS command fails with exception following merge of HADOOP-8325 (tucu) (Revision 1333224)

     Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333224
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java

                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264398#comment-13264398 ] 

Alejandro Abdelnur commented on HADOOP-8325:
--------------------------------------------

Not to mention that a thread is not guaranteed to fully run before another one



                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266974#comment-13266974 ] 

Hudson commented on HADOOP-8325:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #2191 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2191/])
    HADOOP-8342. HDFS command fails with exception following merge of HADOOP-8325 (tucu) (Revision 1333224)

     Result = ABORTED
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333224
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java

                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-8325:
---------------------------------------

    Attachment: HADOOP-8325.patch

retrofitted all uses (in common) of JVM shutdown hook to use ShutdownHookManager
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263951#comment-13263951 ] 

Hadoop QA commented on HADOOP-8325:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12524912/HADOOP-8325.patch
  against trunk revision .

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 2 new or modified test files.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 eclipse:eclipse.  The patch built with eclipse:eclipse.

    -1 findbugs.  The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these unit tests:
                  org.apache.hadoop.io.TestSequenceFile

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/894//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/894//artifact/trunk/hadoop-common-project/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/894//console

This message is automatically generated.
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-8325:
---------------------------------------

    Attachment: HADOOP-8325.patch

re-uploading to see if jenkins picks it up
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-8325:
---------------------------------------

    Attachment: HADOOP-8325.patch

findbugs fixes.

the TestSequenceFile failure seems unrelated as it is also failing on a fresh trunk checkout.
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265134#comment-13265134 ] 

Hadoop QA commented on HADOOP-8325:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12525091/HADOOP-8325.patch
  against trunk revision .

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 2 new or modified test files.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 eclipse:eclipse.  The patch built with eclipse:eclipse.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these unit tests:
                  org.apache.hadoop.io.TestSequenceFile

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/906//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/906//console

This message is automatically generated.
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-8325:
---------------------------------------

    Attachment: HADOOP-8325.patch

Patch integrating last 2 comments from Nicholas.
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HADOOP-8325) Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook

Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264347#comment-13264347 ] 

Alejandro Abdelnur commented on HADOOP-8325:
--------------------------------------------

TestSequenceFile failure seems unrelated.
                
> Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8325
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch, HADOOP-8325.patch
>
>
> FileSystem adds a JVM shutdown hook when a filesystem instance is cached.
> MRAppMaster also uses a JVM shutdown hook, among other things, the MRAppMaster JVM shutdown hook is used to ensure state are written to HDFS.
> This creates a race condition because each JVM shutdown hook is a separate thread and if there are multiple JVM shutdown hooks there is not assurance of order of execution, they could even run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira