[jira] Commented: (HADOOP-1432) Use Grizzly for Fetching Map Output in Shuffle

["Tahir Hashmi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12502742 ] 

Tahir Hashmi commented on HADOOP-1432:
--------------------------------------

News here. Thanks to pushing, shoving and helping from Devaraj, I've got Grizzly to serve map outputs. In terms of high level changes, StatusHttpServer now has a new constructor in addition to the earlier one. The new constructor accepts a class that implements Grizzly's TCP Adapter. Alert! this makes the constructor dependent on Grizzly but then constructors are disposable ;-). If this constructor is called, StatusHttpServer will start Jetty as well as Grizzly. By default, Grizzly will be started on a random port between 10k and 64k -1. In case the port is already taken, port rolling has been provided for and it happens independent of jetty port rolling. The mapOutputPort is a new piece of information which is ultimately communicated to Reduce tasks.

Unit tests pass. RandomWriter on a single node works. You need to apply the attached patch and unpack the attached tarball in src/lib to make things work. Comments invited.

> Use Grizzly for Fetching Map Output in Shuffle
> ----------------------------------------------
>
>                 Key: HADOOP-1432
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1432
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Tahir Hashmi
>            Assignee: Tahir Hashmi
>         Attachments: 1432.patch, grizzly.tgz
>
>
> As mentioned in HADOOP-1273 and references therefrom, Jetty 6 still doesn't seem to be stable enough for use in Hadoop. Instead, we've decided to consider the usage of Grizzly Framework [https://grizzly.dev.java.net/] for NIO based communication.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.