[jira] [Closed] (TOREE-309) pyspark_runner needs to be upgraded for Spark 1.5, 1.6

["Gino Bustelo (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/TOREE-309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gino Bustelo closed TOREE-309.
------------------------------

> pyspark_runner needs to be upgraded for Spark 1.5, 1.6
> ------------------------------------------------------
>
>                 Key: TOREE-309
>                 URL: https://issues.apache.org/jira/browse/TOREE-309
>             Project: TOREE
>          Issue Type: Bug
>    Affects Versions: 0.1.0
>            Reporter: Eric Chang
>             Fix For: 0.1.0
>
>
> pyspark_runner.py includes version-specific switches for Spark 1.4 that should be applied to spark 1.5/1.6:
> {code}
> if sparkVersion.startswith("1.4"):
>   gateway = JavaGateway(client, auto_convert = True)
> else:
>   gateway = JavaGateway(client)
> ...
> elif sparkVersion.startswith("1.4"):
>   java_import(gateway.jvm, "org.apache.spark.sql.*")
>   java_import(gateway.jvm, "org.apache.spark.sql.hive.*")
> {code}
> I have a branch on my fork that extends the 1.4 logic to 1.5, 1.6.  Not sure if packaging is going to change significantly in 2.x so I didn't extend beyond 1.6: https://github.com/apache/incubator-toree/compare/master...ericchang:pyspark-runner-1.5-fix
> Let me know if it's ok for me to issue the PR against the main repo.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)