[GSoC] Map-Reduce Integration with Tuscany

[Chris Trezzo <ct...@gmail.com>]

Hi everyone,

I have been fleshing out a slightly more detailed plan for the  
upcoming weeks, and would like to share it with the community and get  
some feedback.

For the first iteration I am planning to focus on Java implemented Map- 
Reduce (MR) applications. These apps interface directly with Hadoop's  
MR framework[1], as opposed to Hadoop Streaming[2] or Pipes[3].

I think the first priority should be to get a basic MR data flow  
working, and the three necessary entities of a basic MR application  
seem to be the Mapper, the Reducer, and a job configuration. I am  
planning on getting the functionality for these three parts  
implemented first.

Going along with the original design in the proposal, I am planning to  
view the Mapper and the Reducer as implementation types, and the job  
configuration as part of a management layer in charge of the assembly  
and deployment of MR applications. Initially the management layer  
would be responsible for the configuration of MR jobs and the  
integration with Hadoop's MR framework, with the overall goal of  
eventually extending it into something more along the lines of what  
was described by Robert Donkin[4] and Jean-Sebastian (referred to as  
item 3)[5]. In this case, the layer could be used to manage the  
deployment of components over a Hadoop cluster itself.

For the Mapper and Reducer, in the next couple of weeks I would like  
to outline the definition of these types and hopefully start  
implementing them.

For the management layer, I could use some guidance on how to best fit  
it into the Tuscany architectural framework.

Thoughts/Suggestions on any part of my plans are always greatly  
appreciated.

Congrats to everyone for graduation!

Thanks,
Chris Trezzo

[1] http://hadoop.apache.org/core/docs/r0.15.3/api/org/apache/hadoop/mapred/package-summary.html
[2] http://hadoop.apache.org/core/docs/current/streaming.html
[3] http://hadoop.apache.org/core/docs/r0.15.3/api/org/apache/hadoop/mapred/pipes/package-summary.html
[4] http://www.mail-archive.com/tuscany-dev@ws.apache.org/msg29711.html
[5] http://www.mail-archive.com/tuscany-dev@ws.apache.org/msg29720.html