You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@mrql.apache.org by "Leonidas Fegaras (JIRA)" <ji...@apache.org> on 2014/09/23 22:24:33 UTC

[jira] [Created] (MRQL-51) Support for Spark 1.1.0

Leonidas Fegaras created MRQL-51:
------------------------------------

             Summary: Support for Spark 1.1.0
                 Key: MRQL-51
                 URL: https://issues.apache.org/jira/browse/MRQL-51
             Project: MRQL
          Issue Type: Bug
          Components: Run-Time/Spark
    Affects Versions: 0.9.4
            Reporter: Leonidas Fegaras
            Assignee: Leonidas Fegaras
            Priority: Critical


The following patch makes some necessary changes to support Spark 1.1.0. The Spark evaluator now uses RDD.mapPartitionsWithIndex to extract RDD elements lazily (as needed) instead of using the internal Spark method RDD.iterator, which needs access to TaskContext. It also changes the JavaSparkContext to read the Spark config parameters.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)