[jira] [Commented] (DATAFU-167) Fix Scala Python Bridge support in Spark 2 minor version updates

["Arpit Bhardwaj (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DATAFU-167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17618796#comment-17618796 ] 

Arpit Bhardwaj commented on DATAFU-167:
---------------------------------------

[~eyal] I will implemented the changes in branch [https://github.com/arpitbhardwaj/datafu/tree/spark_upgrade_fix]
but i have an open question on regarding this

do we need to branch the old flow without auth and new flow with auth based on spark version? If yes how do we read the client spark version? i have referred apache/zepplin project and there they have done something similar branching.

> Fix Scala Python Bridge support in Spark 2 minor version updates 
> -----------------------------------------------------------------
>
>                 Key: DATAFU-167
>                 URL: https://issues.apache.org/jira/browse/DATAFU-167
>             Project: DataFu
>          Issue Type: Bug
>    Affects Versions: 1.6.1
>         Environment: _emphasized text_
>            Reporter: Eyal Allweil
>            Priority: Major
>              Labels: up-for-grabs
>             Fix For: 1.7.0
>
>
> The Scala Python Bridge, which works for versions 2.2.2, 2.3.2 and 2.4.3, doesn't work for versions 2.2.3, 2.3.3, and 2.4.4 (and up).
> This can be reproduced in testing by running the tests with the command
> {code:bash}
> ./gradlew :datafu-spark:test -PscalaVersion=2.11 -PsparkVersion=2.2.3 -PscalaCompatVersion=2.11
> {code}
> The error message is
> {noformat}
>  AttributeError: 'GatewayParameters' object has no attribute 'auth_token'
> 	at org.apache.spark.datafu.deploy.SparkPythonRunner.execFile(SparkPythonRunner.scala:137)
> {noformat}
>  
> Currently our code runs using the PYSPARK_ALLOW_INSECURE_GATEWAY parameter; it's possible/probable that using the auth_token parameter will both fix this problem and be better in general.
>  
> Please note that in order to test Spark 2.4.4, you need to upgrade the scalatest version used.
>  
> A description of using an auth token with py4j can be found here:
> [https://www.py4j.org/advanced_topics.html#authentication]
>  
> It's likely that (at least some of the) changes will need to be made here:
> [https://github.com/apache/datafu/blob/master/datafu-spark/src/main/scala/spark/utils/overwrites/SparkPythonRunner.scala#L63]
> [https://github.com/apache/datafu/blob/master/datafu-spark/src/main/resources/pyspark_utils/bridge_utils.py#L43]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)