You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by "Dave Beech (JIRA)" <ji...@apache.org> on 2013/06/30 22:43:20 UTC
[jira] [Commented] (CRUNCH-231) Support legacy Mappers and Reducers
in Crunch pipelines
[ https://issues.apache.org/jira/browse/CRUNCH-231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13696408#comment-13696408 ]
Dave Beech commented on CRUNCH-231:
-----------------------------------
My first instinct is not to like this! For me, the higher-level "pipeline" model is much cleaner and nicer to work with than standard mapreduce. Also, mixing different levels of abstraction in one codebase feels like a bad thing to be doing.
Having said all that, I can see the appeal of being able to gradually migrate old code across in terms of lowering the barrier to entry and therefore growing the userbase of Crunch.
>From a quick glance at the patch it looks like just writables are supported. I guess the levels of crazy hacking would need to be stepped up to support Avro -- especially Avro mapred rather than mapreduce!
I'll try and give the patch a go early this week and come back with some more comments.
> Support legacy Mappers and Reducers in Crunch pipelines
> -------------------------------------------------------
>
> Key: CRUNCH-231
> URL: https://issues.apache.org/jira/browse/CRUNCH-231
> Project: Crunch
> Issue Type: Improvement
> Components: Core
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: mapred.patch
>
>
> I've had a few requests for Crunch to support existing Mappers and Reducers using the underlying Java APIs as part of regular pipelines, so that users could evolve existing MapReduce jobs into Crunch pipelines gradually, instead of being forced to rewrite everything all at once in order to map it onto Crunch's model.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira