[jira] [Created] (MAPREDUCE-4876) Adopt a Tuple MapReduce API instead of classic MapReduce one

["Iván de Prado (JIRA)" <ji...@apache.org>]

Iván de Prado created MAPREDUCE-4876:
----------------------------------------

             Summary: Adopt a Tuple MapReduce API instead of classic MapReduce one
                 Key: MAPREDUCE-4876
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4876
             Project: Hadoop Map/Reduce
          Issue Type: Wish
            Reporter: Iván de Prado
            Priority: Minor


After using MapReduce for many years, we have notice that it lacks some important features: compound records, easy intra-reduce sorting and join capabilities. We have elaborated slightly modified MapReduce foundation to overcome these problems: Tuple MapReduce. You can see a full paper published at the ICDM 2012 that describes it at http://pangool.net/TupleMapReduce.pdf 

The good news are:
1) That it is not needed any architectural change on Hadoop to embrace Tuple MapReduce
2) Indeed, we have proven that it is possible to implement it on top of Hadoop. See the Pangool Open Source project ( http://pangool.net/ ). 
3) It performs very efficiently ( http://pangool.net/benchmark.html )
4) It is compatible with all Hadoop stack: Writables, Serializers, Input/OutputFormats, etc. 

We believe Hadoop community could benefits from it in different ways:
1) By getting ideas for future API redesign
2) By adopting Pangool inside Hadoop. Of course, we would be helping and contributing with anything needed, including by doing any adaptation changes needed (too few, because as I told, everything is compatible with existing MapReduce)

Obviously, we prefer the second. But at least, we believe some good ideas can be obtained by looking at Tuple MapReduce and Pangool.  

There are also other improvements in Pangool that would improve Hadoop API:
1) Configuration by instance: passing parameters by constructor. For example, Pangool Input/OutputFormats can be configured by providing values to the constructor
2) Stateful serialization. What is requested in  https://issues.apache.org/jira/browse/MAPREDUCE-1462 is supported by Pangool
3) First-class multipleinput/multipleoutput

Well, we are open to the discussion and to contribute.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira