You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@beam.apache.org by "Amit Sela (JIRA)" <ji...@apache.org> on 2017/04/09 09:08:41 UTC

[jira] [Commented] (BEAM-1920) Add Spark 2.x support in Spark runner

    [ https://issues.apache.org/jira/browse/BEAM-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15962076#comment-15962076 ] 

Amit Sela commented on BEAM-1920:
---------------------------------

Sweet! 
I'm considering Spark 2.x as default and {{1.6.3}} as the profile-enabled version because there are many issues with flaky tests now that I believe that should be resolved as a part of a bunch of bug fixes introduced in Spark {{2.0.0}} and {{2.0.1}}.

The only Con I can think of for that is that Spark 2.x dependency would provide an easy path to new, incompatible, APIs in Spark so we'd have to think carefully how we test for {{1.6.3}} breaking changes.  

> Add Spark 2.x support in Spark runner
> -------------------------------------
>
>                 Key: BEAM-1920
>                 URL: https://issues.apache.org/jira/browse/BEAM-1920
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-spark
>            Reporter: Jean-Baptiste Onofré
>            Assignee: Jean-Baptiste Onofré
>
> I have a branch working with both Spark 1 and Spark 2 backend.
> I'm preparing a pull request about that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)