You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Sharad Agarwal (JIRA)" <ji...@apache.org> on 2009/07/23 12:31:14 UTC

[jira] Commented: (MAPREDUCE-786) Jobtracker history should be written aysnchronously to the filesystem

    [ https://issues.apache.org/jira/browse/MAPREDUCE-786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734542#action_12734542 ] 

Sharad Agarwal commented on MAPREDUCE-786:
------------------------------------------

I think we should cleanup the way history is written. Providing cleaner API to plugin the async. Perhaps a separate JIRA for it. Something like:
{code}
public interface Event {
  public Map<String, String> getData();
}

public interface EventReader {
   public Iterator<Event> read();
   public void close();
}

public interface EventWriter {
  public void write(Event event);
  public void flush();
  public void close();
}
{code}

Async writer should be able to wrap to an EventWriter object:
{code}
class AsyncEventWriter implements EventWriter {
  AsyncEventWriter(EventWriter writer){
  }
}
{code}

> Jobtracker history should be written aysnchronously to the filesystem
> ---------------------------------------------------------------------
>
>                 Key: MAPREDUCE-786
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-786
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>
> Jobtracker lock is held while writing the history events. This makes the jobtracker slow on flushes, especially when history is written to HDFS. History events should be written asynchronously to avoid this problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.