You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@datafu.apache.org by "Eyal Allweil (Jira)" <ji...@apache.org> on 2022/10/11 13:05:00 UTC
[jira] [Updated] (DATAFU-167) Fix Scala Python Bridge support in Spark 2 minor version updates
[ https://issues.apache.org/jira/browse/DATAFU-167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eyal Allweil updated DATAFU-167:
--------------------------------
Description:
The Scala Python Bridge, which works for versions 2.2.2, 2.3.2 and 2.4.3, doesn't work for versions 2.2.3, 2.3.3, and 2.4.4 (and up).
This can be reproduced in testing by running the tests with the command
{code:bash}
./gradlew :datafu-spark:test -PscalaVersion=2.11 -PsparkVersion=2.2.3 -PscalaCompatVersion=2.11
{code}
The error message is
{noformat}
AttributeError: 'GatewayParameters' object has no attribute 'auth_token'
at org.apache.spark.datafu.deploy.SparkPythonRunner.execFile(SparkPythonRunner.scala:137)
{noformat}
Currently our code runs using the PYSPARK_ALLOW_INSECURE_GATEWAY parameter; it's possible/probable that using the auth_token parameter will both fix this problem and be better in general.
Please note that in order to test Spark 2.4.4, you need to upgrade the scalatest version used.
was:
The Scala Python Bridge, which works for versions 2.2.2, 2.3.2 and 2.4.3, doesn't work for versions 2.2.3, 2.3.3, and 2.4.4 (and up).
This can be reproduced in testing by running the tests with the command
{code:bash}
./gradlew :datafu-spark:test -PscalaVersion=2.11 -PsparkVersion=2.2.3 -PscalaCompatVersion=2.11
{code}
The error message is
{noformat}
AttributeError: 'GatewayParameters' object has no attribute 'auth_token'
at org.apache.spark.datafu.deploy.SparkPythonRunner.execFile(SparkPythonRunner.scala:137)
{noformat}
Currently our code runs using the PYSPARK_ALLOW_INSECURE_GATEWAY parameter; it's possible/probable that using the auth_token parameter will both fix this problem and be better in general.
> Fix Scala Python Bridge support in Spark 2 minor version updates
> -----------------------------------------------------------------
>
> Key: DATAFU-167
> URL: https://issues.apache.org/jira/browse/DATAFU-167
> Project: DataFu
> Issue Type: Bug
> Affects Versions: 1.6.1
> Environment: _emphasized text_
> Reporter: Eyal Allweil
> Priority: Major
> Fix For: 1.7.0
>
>
> The Scala Python Bridge, which works for versions 2.2.2, 2.3.2 and 2.4.3, doesn't work for versions 2.2.3, 2.3.3, and 2.4.4 (and up).
> This can be reproduced in testing by running the tests with the command
> {code:bash}
> ./gradlew :datafu-spark:test -PscalaVersion=2.11 -PsparkVersion=2.2.3 -PscalaCompatVersion=2.11
> {code}
> The error message is
> {noformat}
> AttributeError: 'GatewayParameters' object has no attribute 'auth_token'
> at org.apache.spark.datafu.deploy.SparkPythonRunner.execFile(SparkPythonRunner.scala:137)
> {noformat}
>
> Currently our code runs using the PYSPARK_ALLOW_INSECURE_GATEWAY parameter; it's possible/probable that using the auth_token parameter will both fix this problem and be better in general.
>
> Please note that in order to test Spark 2.4.4, you need to upgrade the scalatest version used.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)