You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@crunch.apache.org by "Josh Wills (JIRA)" <ji...@apache.org> on 2014/04/29 18:50:15 UTC

[jira] [Updated] (CRUNCH-384) Upgrade Spark to 0.9.1

     [ https://issues.apache.org/jira/browse/CRUNCH-384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Josh Wills updated CRUNCH-384:
------------------------------

    Attachment: CRUNCH-384.patch

This patch upgrades Crunch-on-Spark to use Spark 0.9.1 and fixes a number of limitations in the old version: this version works w/Spark in standalone mode on top of HDFS and also makes using Counters transparently map to using Spark's accumulators (even storing them in the same place in the PipelineResult object.)

[~mkwhitacre] want to take a look at this one?

> Upgrade Spark to 0.9.1
> ----------------------
>
>                 Key: CRUNCH-384
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-384
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Josh Wills
>            Assignee: Josh Wills
>         Attachments: CRUNCH-384.patch
>
>
> Spark 0.9.x has better compatibility with Hadoop 2.x along with a bunch of other fixes. It also requires Scala 2.10, which we should upgrade to anyway.



--
This message was sent by Atlassian JIRA
(v6.2#6252)