You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@crunch.apache.org by "Rafal Wojdyla (JIRA)" <ji...@apache.org> on 2014/09/11 12:27:33 UTC

[jira] [Created] (CRUNCH-470) Add hdfs/yarn minicluster crunch pipeline

Rafal Wojdyla created CRUNCH-470:
------------------------------------

             Summary: Add hdfs/yarn minicluster crunch pipeline
                 Key: CRUNCH-470
                 URL: https://issues.apache.org/jira/browse/CRUNCH-470
             Project: Crunch
          Issue Type: Improvement
          Components: Core
    Affects Versions: 0.8.3
            Reporter: Rafal Wojdyla
            Assignee: Josh Wills
            Priority: Minor


Crunch currently has two pipelines:
* MemPipeline
* MRPipeline

MemPipeline is in-memory pipelines based on local in-memory mapreduce mode.
MRPipeline is distributed pipeline based on distributed MapReduce.

Using HDFS/YARN Minicluster it's possible to better emulate Hadoop cluster, and it could be a 'final test' before running on the cluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)