[jira] [Resolved] (MAPREDUCE-5588) TaskTrackers get killed by JettyBugMonitor because of incredibly high cpu usage

["Chris Nauroth (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAPREDUCE-5588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Nauroth resolved MAPREDUCE-5588.
--------------------------------------

    Resolution: Duplicate
      Assignee: Chris Nauroth

Hello, [~moian].

There is a known bug in Jetty that can cause it to consume a ton of CPU and not make any real progress.  As a workaround, the Hadoop code monitors for this condition and kills the tasktracker process if it finds Jetty consuming too much CPU.  (It's better to kill the tasktracker outright then leave it running in an unresponsive state.)  This monitoring thread was added in MAPREDUCE-3184.  If you review that issue, you can find more background information on configuration and tuning.

The bug is set to be fixed in Jetty version 6.1.27, but we've been waiting a long time on that release.  We also can't easily jump to a new major version of Jetty due to API backwards-compatibility issues.  MAPREDUCE-2980 contains ongoing discussion about our upgrade plan.  As a workaround, there are also patches available that you can apply on top of Jetty 6.1.26.

I'm going to resolve this as a duplicate of MAPREDUCE-2980.

> TaskTrackers get killed by JettyBugMonitor because of incredibly high cpu usage
> -------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5588
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5588
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 1.1.2
>            Reporter: Anthony MOI
>            Assignee: Chris Nauroth
>              Labels: cpu-usage, jetty, tasktracker
>
> We are running a little cluster with 10 servers running task trackers. All of them are getting killed randomly with the following message
> {quote}
> 2013-10-17 11:32:31,037 FATAL org.apache.hadoop.mapred.JettyBugMonitor: ************************************************************
> Jetty CPU usage: 120093277.1%. This is greater than the fatal threshold mapred.tasktracker.jetty.cpu.threshold.fatal. Aborting JVM.
> ************************************************************
> 2013-10-17 11:32:31,039 INFO org.apache.hadoop.mapred.TaskTracker: SHUTDOWN_MSG: 
> /************************************************************
> SHUTDOWN_MSG: Shutting down TaskTracker
> ************************************************************/
> {quote}
> Everytime, the message notices a cpu usage above 120M%. Everything has been running for a while now (since 1.1.2 release) without any problems, and it started just like that.
> Any idea of what could cause this ?



--
This message was sent by Atlassian JIRA
(v6.1#6144)