You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hadoop.apache.org by foolbear <fo...@gmail.com> on 2015/01/15 05:26:55 UTC

Does YARN has hash-based shuffle plugin?

Hi

In YARN, shuffle and sort is pluggable:
http://hadoop.apache.org/docs/r2.5.2/hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html

Currently, shuffle is based on sort. But many of my mapreduce jobs do not
need sort.
To improve performance, maybe it is better to avoid sort and use hash
instead.

So, is there a hash-based shuffle plugin?
Seems hadoop itselt does not do this. Any third-party implements?

Thanks