You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@toree.apache.org by "Gino Bustelo (JIRA)" <ji...@apache.org> on 2016/04/28 14:32:12 UTC
[jira] [Commented] (TOREE-309) pyspark_runner needs to be upgraded
for Spark 1.5, 1.6
[ https://issues.apache.org/jira/browse/TOREE-309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262058#comment-15262058 ]
Gino Bustelo commented on TOREE-309:
------------------------------------
PR are the way to go. Please submit them for both issues you opened.
> pyspark_runner needs to be upgraded for Spark 1.5, 1.6
> ------------------------------------------------------
>
> Key: TOREE-309
> URL: https://issues.apache.org/jira/browse/TOREE-309
> Project: TOREE
> Issue Type: Bug
> Affects Versions: 0.1.0
> Reporter: Eric Chang
>
> pyspark_runner.py includes version-specific switches for Spark 1.4 that should be applied to spark 1.5/1.6:
> {code}
> if sparkVersion.startswith("1.4"):
> gateway = JavaGateway(client, auto_convert = True)
> else:
> gateway = JavaGateway(client)
> ...
> elif sparkVersion.startswith("1.4"):
> java_import(gateway.jvm, "org.apache.spark.sql.*")
> java_import(gateway.jvm, "org.apache.spark.sql.hive.*")
> {code}
> I have a branch on my fork that extends the 1.4 logic to 1.5, 1.6. Not sure if packaging is going to change significantly in 2.x so I didn't extend beyond 1.6: https://github.com/apache/incubator-toree/compare/master...ericchang:pyspark-runner-1.5-fix
> Let me know if it's ok for me to issue the PR against the main repo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)