You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Devaraj K (Created) (JIRA)" <ji...@apache.org> on 2012/04/11 08:04:28 UTC
[jira] [Created] (MAPREDUCE-4135) MRAppMaster throws
IllegalStateException while shutting down
MRAppMaster throws IllegalStateException while shutting down
------------------------------------------------------------
Key: MAPREDUCE-4135
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 3.0.0
Reporter: Devaraj K
Always MRAppMaster throws IllegalStateException in the stderr while shutting down. It doesn't look good having this exception in the stderr file of MRAppMaster container.
{code:xml}
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in progress
at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws
IllegalStateException while shutting down
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252253#comment-13252253 ]
Alejandro Abdelnur commented on MAPREDUCE-4135:
-----------------------------------------------
+1
> MRAppMaster throws IllegalStateException while shutting down
> ------------------------------------------------------------
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0, 3.0.0
> Reporter: Devaraj K
> Assignee: Devaraj K
> Attachments: MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting down. It doesn't look good having this exception in the stderr file of MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in progress
> at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws
IllegalStateException while shutting down
Posted by "Ravi Prakash (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264350#comment-13264350 ]
Ravi Prakash commented on MAPREDUCE-4135:
-----------------------------------------
Thanks Tucu! I hadn't seen HADOOP-8325. You probably meant MAPREDUCE-4157. I am planning on letting this JIRA stay open until either / both of those JIRA make the problem go away.
> MRAppMaster throws IllegalStateException while shutting down
> ------------------------------------------------------------
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0, 3.0.0
> Reporter: Devaraj K
> Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4135.branch-0.23.patch, MAPREDUCE-4135.patch, MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting down. It doesn't look good having this exception in the stderr file of MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in progress
> at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws
IllegalStateException while shutting down
Posted by "Ravi Prakash (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261823#comment-13261823 ]
Ravi Prakash commented on MAPREDUCE-4135:
-----------------------------------------
I'm sorry I was wrong about my earlier statement:
bq. This is causing the subsequent statement (FileSystem.closeAll() ) to be skipped. So to fix this JIRA, I'll see what I can do to fix the InterruptedException.
The InterruptedException is thrown and then caught. So its not that. Its MAPREDUCE-4157. The SIGTERM is sent without any delay on ContainerLaunch.java:337.
> MRAppMaster throws IllegalStateException while shutting down
> ------------------------------------------------------------
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0, 3.0.0
> Reporter: Devaraj K
> Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4135.branch-0.23.patch, MAPREDUCE-4135.patch, MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting down. It doesn't look good having this exception in the stderr file of MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in progress
> at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4135) MRAppMaster throws
IllegalStateException while shutting down
Posted by "Ravi Prakash (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Prakash updated MAPREDUCE-4135:
------------------------------------
Attachment: MAPREDUCE-4135.branch-0.23.patch
Wait! I thought of another way.
3. We can @Override stop() in MRAppMaster to call super.stop() and then do the FileSystem.closeAll(). Set a flag to denote that stop() has already been called. In the MRAppMasterShutdownHook, if the flag is set, NOT call the FileSystem.closeAll(). Sheesh! Now I feel stupid for not thinking about this earlier :(
What do all y'all think of this way?
> MRAppMaster throws IllegalStateException while shutting down
> ------------------------------------------------------------
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0, 3.0.0
> Reporter: Devaraj K
> Assignee: Devaraj K
> Attachments: MAPREDUCE-4135.branch-0.23.patch, MAPREDUCE-4135.patch, MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting down. It doesn't look good having this exception in the stderr file of MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in progress
> at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws
IllegalStateException while shutting down
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263969#comment-13263969 ]
Alejandro Abdelnur commented on MAPREDUCE-4135:
-----------------------------------------------
HADOOP-8325 & MAPREDUCE-4135 take care of this problem in a more generic way. Please evaluate and close if so.
> MRAppMaster throws IllegalStateException while shutting down
> ------------------------------------------------------------
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0, 3.0.0
> Reporter: Devaraj K
> Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4135.branch-0.23.patch, MAPREDUCE-4135.patch, MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting down. It doesn't look good having this exception in the stderr file of MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in progress
> at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4135) MRAppMaster throws
IllegalStateException while shutting down
Posted by "Ravi Prakash (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Prakash reassigned MAPREDUCE-4135:
---------------------------------------
Assignee: Ravi Prakash (was: Devaraj K)
> MRAppMaster throws IllegalStateException while shutting down
> ------------------------------------------------------------
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0, 3.0.0
> Reporter: Devaraj K
> Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4135.branch-0.23.patch, MAPREDUCE-4135.patch, MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting down. It doesn't look good having this exception in the stderr file of MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in progress
> at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4135) MRAppMaster throws
IllegalStateException while shutting down
Posted by "Devaraj K (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-4135:
---------------------------------
Status: Open (was: Patch Available)
> MRAppMaster throws IllegalStateException while shutting down
> ------------------------------------------------------------
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0, 3.0.0
> Reporter: Devaraj K
> Assignee: Devaraj K
> Attachments: MAPREDUCE-4135.patch, MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting down. It doesn't look good having this exception in the stderr file of MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in progress
> at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws
IllegalStateException while shutting down
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263104#comment-13263104 ]
Alejandro Abdelnur commented on MAPREDUCE-4135:
-----------------------------------------------
First, I don't think FileSystem.closeAll() should try to remove its shutdown hook, the hook could be there and it would just be a NOP.
Secondly, shutdown hooks are run in parallel/non-deterministic-order. if there is nothing that ensures that the FileSystem shutdown hook won't be run until after the MRAppMaster shutdown hook.
IMO the correct solution here would be:
* All close/destroy/cleanup operations are done only via Service's stop() method. This ensures an order.
* The outermost service is the only one adding a shutdown hook and within it i call its stop() which will do a NOP if it has been already called.
> MRAppMaster throws IllegalStateException while shutting down
> ------------------------------------------------------------
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0, 3.0.0
> Reporter: Devaraj K
> Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4135.branch-0.23.patch, MAPREDUCE-4135.patch, MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting down. It doesn't look good having this exception in the stderr file of MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in progress
> at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws
IllegalStateException while shutting down
Posted by "Devaraj K (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252355#comment-13252355 ]
Devaraj K commented on MAPREDUCE-4135:
--------------------------------------
These above test failures are not related to the patch. Seems these are random failures with the reason "java.net.BindException: Problem binding to [0.0.0.0:8031] java.net.BindException: Address already in use". Same tests failed for MAPREDUCE-4140 QA also.
> MRAppMaster throws IllegalStateException while shutting down
> ------------------------------------------------------------
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0, 3.0.0
> Reporter: Devaraj K
> Assignee: Devaraj K
> Attachments: MAPREDUCE-4135.patch, MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting down. It doesn't look good having this exception in the stderr file of MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in progress
> at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws
IllegalStateException while shutting down
Posted by "Ravi Prakash (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260614#comment-13260614 ]
Ravi Prakash commented on MAPREDUCE-4135:
-----------------------------------------
Hi Devaraj! I've assigned this JIRA to myself. I hope you don't mind.
AFTER applying the latest patch, inside MRAppMaster.stop(), super.stop() is throwing an InterruptedException
{noformat}
2012-04-24 09:25:20,104 WARN [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Interrupted Exception while stopping
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1186)
at java.lang.Thread.join(Thread.java:1239)
at org.apache.hadoop.yarn.event.AsyncDispatcher.stop(AsyncDispatcher.java:104)
at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:99)
at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1016)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:418)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:386)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:74)
at java.lang.Thread.run(Thread.java:662)
{noformat}
This is causing the subsequent statement (FileSystem.closeAll() ) to be skipped. So to fix this JIRA, I'll see what I can do to fix the InterruptedException.
> MRAppMaster throws IllegalStateException while shutting down
> ------------------------------------------------------------
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0, 3.0.0
> Reporter: Devaraj K
> Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4135.branch-0.23.patch, MAPREDUCE-4135.patch, MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting down. It doesn't look good having this exception in the stderr file of MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in progress
> at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws
IllegalStateException while shutting down
Posted by "Ravi Prakash (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252608#comment-13252608 ]
Ravi Prakash commented on MAPREDUCE-4135:
-----------------------------------------
This is a particularly vexing problem. :( I can think of two ways out of this
1. Swallow IllegalStateExceptions thrown by removeShutdownHook in FileSystem. The javadoc states:
{quote}public boolean removeShutdownHook(Thread hook)
Throws:
IllegalStateException - If the virtual machine is already in the process of shutting down {quote}
So if we are getting this exception, it MEANS we are already in the process of shutdown, so we CANNOT, try what we may, removeShutdownHook. If Runtime had a method Runtime.isShutdownInProgress(), we could have checked for it before the removeShutdownHook call. As it stands, there is no such method. In my opinion, this would be a good patch regardless of the needs for this JIRA.
2. Not send SIGTERMs from the NM to the MR-AM in the first place. Rather we should expose a mechanism for the NM to politely tell the AM its no longer needed and should shutdown asap. Even after this, if an admin were to kill the MRAppMaster with a SIGTERM, the JobHistory would be lost defeating the purpose of 3614
> MRAppMaster throws IllegalStateException while shutting down
> ------------------------------------------------------------
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0, 3.0.0
> Reporter: Devaraj K
> Assignee: Devaraj K
> Attachments: MAPREDUCE-4135.patch, MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting down. It doesn't look good having this exception in the stderr file of MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in progress
> at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws
IllegalStateException while shutting down
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254830#comment-13254830 ]
Hadoop QA commented on MAPREDUCE-4135:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522790/MAPREDUCE-4135.branch-0.23.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell
org.apache.hadoop.yarn.server.TestDiskFailures
org.apache.hadoop.yarn.server.TestContainerManagerSecurity
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
org.apache.hadoop.mapred.TestMiniMRClasspath
org.apache.hadoop.mapreduce.v2.TestMRJobs
org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
org.apache.hadoop.mapred.TestMiniMRBringup
org.apache.hadoop.mapred.TestMiniMRChildTask
org.apache.hadoop.mapred.TestReduceFetch
org.apache.hadoop.mapred.TestClusterMRNotification
org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
org.apache.hadoop.mapred.TestJobCounters
org.apache.hadoop.mapreduce.TestChild
org.apache.hadoop.mapred.TestMiniMRClientCluster
org.apache.hadoop.ipc.TestSocketFactory
org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
org.apache.hadoop.mapred.TestClientRedirect
org.apache.hadoop.mapred.TestLazyOutput
org.apache.hadoop.mapred.TestJobCleanup
org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner
org.apache.hadoop.conf.TestNoDefaultsJobConf
org.apache.hadoop.mapreduce.v2.TestRMNMInfo
org.apache.hadoop.mapred.TestClusterMapReduceTestCase
org.apache.hadoop.mapreduce.v2.TestNonExistentJob
org.apache.hadoop.mapred.TestJobSysDirWithDFS
org.apache.hadoop.mapreduce.v2.TestUberAM
org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
org.apache.hadoop.mapred.TestJobName
org.apache.hadoop.mapreduce.security.TestJHSSecurity
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2225//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2225//console
This message is automatically generated.
> MRAppMaster throws IllegalStateException while shutting down
> ------------------------------------------------------------
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0, 3.0.0
> Reporter: Devaraj K
> Assignee: Devaraj K
> Attachments: MAPREDUCE-4135.branch-0.23.patch, MAPREDUCE-4135.patch, MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting down. It doesn't look good having this exception in the stderr file of MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in progress
> at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4135) MRAppMaster throws
IllegalStateException while shutting down
Posted by "Devaraj K (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K reassigned MAPREDUCE-4135:
------------------------------------
Assignee: Devaraj K
> MRAppMaster throws IllegalStateException while shutting down
> ------------------------------------------------------------
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 3.0.0
> Reporter: Devaraj K
> Assignee: Devaraj K
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting down. It doesn't look good having this exception in the stderr file of MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in progress
> at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4135) MRAppMaster throws
IllegalStateException while shutting down
Posted by "Devaraj K (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-4135:
---------------------------------
Status: Patch Available (was: Open)
Re-attaching the same patch to run Hadoop QA.
> MRAppMaster throws IllegalStateException while shutting down
> ------------------------------------------------------------
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0, 3.0.0
> Reporter: Devaraj K
> Assignee: Devaraj K
> Attachments: MAPREDUCE-4135.patch, MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting down. It doesn't look good having this exception in the stderr file of MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in progress
> at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws
IllegalStateException while shutting down
Posted by "Steve Loughran (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251441#comment-13251441 ]
Steve Loughran commented on MAPREDUCE-4135:
-------------------------------------------
The problem is that the FS shutdown process is being triggered by the app master shutdown hook -at which point it is too late to remove other shutdown hooks.
{{FileSystem.Cache.remove()}} should just catch any {{IllegalStateException}} and not worry so much about shutdown hook state failures as the system is usually in an odd state at this point.
> MRAppMaster throws IllegalStateException while shutting down
> ------------------------------------------------------------
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 3.0.0
> Reporter: Devaraj K
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting down. It doesn't look good having this exception in the stderr file of MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in progress
> at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4135) MRAppMaster throws
IllegalStateException while shutting down
Posted by "Ravi Prakash (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Prakash resolved MAPREDUCE-4135.
-------------------------------------
Resolution: Duplicate
Fix Version/s: 0.23.3
Thanks Tucu! HADOOP-8325 seems to have fixed this issue. Duping this jira
> MRAppMaster throws IllegalStateException while shutting down
> ------------------------------------------------------------
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0-alpha, 3.0.0
> Reporter: Devaraj K
> Assignee: Ravi Prakash
> Fix For: 0.23.3
>
> Attachments: MAPREDUCE-4135.branch-0.23.patch, MAPREDUCE-4135.patch, MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting down. It doesn't look good having this exception in the stderr file of MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in progress
> at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4135) MRAppMaster throws
IllegalStateException while shutting down
Posted by "Devaraj K (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-4135:
---------------------------------
Attachment: MAPREDUCE-4135.patch
> MRAppMaster throws IllegalStateException while shutting down
> ------------------------------------------------------------
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0, 3.0.0
> Reporter: Devaraj K
> Assignee: Devaraj K
> Attachments: MAPREDUCE-4135.patch, MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting down. It doesn't look good having this exception in the stderr file of MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in progress
> at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4135) MRAppMaster throws
IllegalStateException while shutting down
Posted by "Ravi Prakash (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Prakash updated MAPREDUCE-4135:
------------------------------------
Status: Open (was: Patch Available)
> MRAppMaster throws IllegalStateException while shutting down
> ------------------------------------------------------------
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0, 3.0.0
> Reporter: Devaraj K
> Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4135.branch-0.23.patch, MAPREDUCE-4135.patch, MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting down. It doesn't look good having this exception in the stderr file of MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in progress
> at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws
IllegalStateException while shutting down
Posted by "Ravi Prakash (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252472#comment-13252472 ]
Ravi Prakash commented on MAPREDUCE-4135:
-----------------------------------------
I'd set automatic close to false to fix MAPREDUCE-3614.
MRAppMaster.java:1000-1003
{noformat}
// Do not automatically close FileSystem objects so that in case of
// SIGTERM I have a chance to write out the job history. I'll be closing
// the objects myself.
conf.setBoolean("fs.automatic.close", false);
{noformat}
We cannot let FileSystem close the handle before we write out JobHistory. We'll have to find a way to still gracefully release all resources.
> MRAppMaster throws IllegalStateException while shutting down
> ------------------------------------------------------------
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0, 3.0.0
> Reporter: Devaraj K
> Assignee: Devaraj K
> Attachments: MAPREDUCE-4135.patch, MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting down. It doesn't look good having this exception in the stderr file of MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in progress
> at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws
IllegalStateException while shutting down
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252341#comment-13252341 ]
Hadoop QA commented on MAPREDUCE-4135:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522393/MAPREDUCE-4135.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell
org.apache.hadoop.yarn.server.TestDiskFailures
org.apache.hadoop.yarn.server.TestContainerManagerSecurity
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
org.apache.hadoop.mapred.TestMiniMRClasspath
org.apache.hadoop.mapreduce.v2.TestMRJobs
org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
org.apache.hadoop.mapred.TestMiniMRBringup
org.apache.hadoop.mapred.TestMiniMRChildTask
org.apache.hadoop.mapred.TestReduceFetch
org.apache.hadoop.mapred.TestClusterMRNotification
org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
org.apache.hadoop.mapred.TestJobCounters
org.apache.hadoop.mapreduce.TestChild
org.apache.hadoop.mapred.TestMiniMRClientCluster
org.apache.hadoop.ipc.TestSocketFactory
org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
org.apache.hadoop.mapred.TestClientRedirect
org.apache.hadoop.mapred.TestLazyOutput
org.apache.hadoop.mapred.TestJobCleanup
org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner
org.apache.hadoop.conf.TestNoDefaultsJobConf
org.apache.hadoop.mapreduce.v2.TestRMNMInfo
org.apache.hadoop.mapred.TestClusterMapReduceTestCase
org.apache.hadoop.mapreduce.v2.TestNonExistentJob
org.apache.hadoop.mapred.TestJobSysDirWithDFS
org.apache.hadoop.mapreduce.v2.TestUberAM
org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
org.apache.hadoop.mapred.TestJobName
org.apache.hadoop.mapreduce.security.TestJHSSecurity
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2209//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2209//console
This message is automatically generated.
> MRAppMaster throws IllegalStateException while shutting down
> ------------------------------------------------------------
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0, 3.0.0
> Reporter: Devaraj K
> Assignee: Devaraj K
> Attachments: MAPREDUCE-4135.patch, MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting down. It doesn't look good having this exception in the stderr file of MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in progress
> at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4135) MRAppMaster throws
IllegalStateException while shutting down
Posted by "Devaraj K (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-4135:
---------------------------------
Attachment: MAPREDUCE-4135.patch
> MRAppMaster throws IllegalStateException while shutting down
> ------------------------------------------------------------
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 3.0.0
> Reporter: Devaraj K
> Assignee: Devaraj K
> Attachments: MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting down. It doesn't look good having this exception in the stderr file of MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in progress
> at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4135) MRAppMaster throws
IllegalStateException while shutting down
Posted by "Devaraj K (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-4135:
---------------------------------
Affects Version/s: 2.0.0
Status: Patch Available (was: Open)
As Alejandro mentioned in MAPREDUCE-4137, Calling FileSystem.closeAll() from MRAppMaster is not required because the FileSystem has its own shutdown hook ClientFinalizer which does closeAll().
I have removed it and attached the patch. I have not added any tests because it is just code removal and verified it manually.
> MRAppMaster throws IllegalStateException while shutting down
> ------------------------------------------------------------
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0, 3.0.0
> Reporter: Devaraj K
> Assignee: Devaraj K
> Attachments: MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting down. It doesn't look good having this exception in the stderr file of MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in progress
> at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira