You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@linkis.apache.org by GitBox <gi...@apache.org> on 2022/04/18 06:47:43 UTC
[GitHub] [incubator-linkis] beidouyunyun opened a new issue, #1991: [Bug] Modify the Spark version to 2.3.0 and compile with error
beidouyunyun opened a new issue, #1991:
URL: https://github.com/apache/incubator-linkis/issues/1991
### Search before asking
- [X] I searched the [issues](https://github.com/apache/incubator-linkis/issues) and found no similar issues.
### Linkis Component
linkis-cg-engineConnplugin
### What happened + What you expected to happen
操作如下：
1.修改incubator-linkis-release-1.0.3/linkis-engineconn-plugins/engineconn-plugins/spark/pom.xml
<spark.version>2.3.0</spark.version>
2.编译linkis-engineconn-plugins
报错内容如下：
[ERROR] /opt/WedataSphere/incubator-linkis-release-1.0.3/linkis-engineconn-plugins/engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/executor/SparkPythonExecutor.scala:87: value authToken is not a member of py4j.GatewayServer.GatewayServerBuilder
[ERROR] try builder.authToken(py4jToken) catch {
[ERROR] ^
[WARNING] /opt/WedataSphere/incubator-linkis-release-1.0.3/linkis-engineconn-plugins/engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/executor/SparkScalaExecutor.scala:96: a pure expression does nothing in statement position; you may be omitting necessary parentheses
[WARNING] null
[WARNING] ^
[WARNING] one warning found
### Relevent platform
Maven == 3.6.3
JDK == 1.8.0_301
Linux == Centos7.8
### Reproduction script
mvn clean install -Dmaven.test.skip=true
### Anything else
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For additional commands, e-mail: dev-help@linkis.apache.org
[GitHub] [incubator-linkis] peacewong closed issue #1991: [Bug] Modify the Spark version to 2.3.0 and compile with error
Posted by GitBox <gi...@apache.org>.
peacewong closed issue #1991: [Bug] Modify the Spark version to 2.3.0 and compile with error
URL: https://github.com/apache/incubator-linkis/issues/1991
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [incubator-linkis] casionone commented on issue #1991: [Bug] Modify the Spark version to 2.3.0 and compile with error
Posted by GitBox <gi...@apache.org>.
casionone commented on issue #1991:
URL: https://github.com/apache/incubator-linkis/issues/1991#issuecomment-1102361240
Unfortunately there is a compatibility issue with spark versions.
The version of net.sf.py4j that spark-core_2.11:2.4.3 depends on is 0.10.7
The version of net.sf.py4j that spark-core_2.11:2.3.0 depends on is 0.10.6
The authToken method was introduced in the py4j.GatewayServer.GatewayServerBuilder version 0.10.7 of py4j.
Details can be viewed:
https://github.com/py4j/py4j/blob/0.10.6/py4j-java/src/main/java/py4j/GatewayServer.java
https://github.com/py4j/py4j/blob/0.10.7/py4j-java/src/main/java/py4j/GatewayServer.java
You can try to manually introduce the dependencies of the 0.10.7 version of py4j to solve this problem
----
是存在spark版本的兼容性问题。
spark-core_2.11:2.4.3版本依赖的net.sf.py4j版本是0.10.7
spark-core_2.11:2.3.0版本依赖的net.sf.py4j版本是0.10.6
py4j的版本0.10.7版本的py4j.GatewayServer.GatewayServerBuilder 才引入了authToken方法 。
详细可以查看：
https://github.com/py4j/py4j/blob/0.10.6/py4j-java/src/main/java/py4j/GatewayServer.java
https://github.com/py4j/py4j/blob/0.10.7/py4j-java/src/main/java/py4j/GatewayServer.java
可以尝试手动引入py4j的0.10.7 版本的依赖，来解决这个问题
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For additional commands, e-mail: dev-help@linkis.apache.org