You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by "Harsh J (Reopened) (JIRA)" <ji...@apache.org> on 2012/03/23 19:21:29 UTC

[jira] [Reopened] (MAPREDUCE-3969) ConcurrentModificationException in JobHistory.java

     [ https://issues.apache.org/jira/browse/MAPREDUCE-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J reopened MAPREDUCE-3969:
--------------------------------

      Assignee: Harsh J

Arun,

Did you get a chance to check the issue before closing this? This is very well present in branch-1 too. Reported version was set to 0.20.2 but it was CDH3 per his environment field, which shares much of the same security code, making it worthwhile to at least take a look at.

Its very rare to hit this, but it is present and it is pretty critical once it hits (and requires a specific TT to restart to resume the tasks, else the job just hangs).

The fix would be to synchronize the writers list object before entering the loop that can potentially modify the list, inside of JobHistory.log(…):

{code}
+ synchronized (writers) {
      for (Iterator<PrintWriter> iter = writers.iterator(); iter.hasNext();) {
        PrintWriter out = iter.next();
        out.println(builder.toString());
        if (out.checkError() && id != null) {
          LOG.info("Logging failed for job " + id + " removing PrintWriter from FileManager");
          iter.remove();
        }
      }
+    }
{code}

Its the {{iter.remove();}} conditional call that causes the issue, as it is done without a lock (while others may access the same set of writers in parallel - the fact which makes this doubly very rare to hit).

Do you agree with the above analysis Arun? If so, I'll post up a patch. Else lemme know what I've missed here.
                
> ConcurrentModificationException in JobHistory.java
> --------------------------------------------------
>
>                 Key: MAPREDUCE-3969
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3969
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>         Environment: cdh3u1 hadoop distribution, centos 5.5.
>            Reporter: Alexey Zotov
>            Assignee: Harsh J
>
> {code}
> 2012-03-01 04:24:47,479 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201202150320_3709_m_000148_0' has completed task_201202150320_3709_m_000148 successfully.
> 2012-03-01 04:24:47,479 INFO org.apache.hadoop.mapred.JobHistory: Logging failed for job job_201202150320_3709removing PrintWriter from FileManager
> 2012-03-01 04:24:47,479 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 8021, call heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@61069281, false, false, true, -21317) from <TASKTRACKER-IP>:450
> 41: error: java.io.IOException: java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
>         at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
>         at java.util.AbstractList$Itr.next(AbstractList.java:343)
>         at org.apache.hadoop.mapred.JobHistory.log(JobHistory.java:591)
>         at org.apache.hadoop.mapred.JobHistory$MapAttempt.logFinished(JobHistory.java:1735)
>         at org.apache.hadoop.mapred.JobInProgress.completedTask(JobInProgress.java:2515)
>         at org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:1200)
>         at org.apache.hadoop.mapred.JobTracker.updateTaskStatuses(JobTracker.java:4539)
>         at org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3503)
>         at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3202)
>         at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
> {code}
> Task task_201202150320_3709_m_000148 was being marked as failed (but had not been restarted) at that moment and job execution was being freezed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira