You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@linkis.apache.org by GitBox <gi...@apache.org> on 2022/01/01 09:04:30 UTC
[GitHub] [incubator-linkis] Zosimer opened a new pull request #1264: Pyspark does not support CDH and HDP versions
Zosimer opened a new pull request #1264:
URL: https://github.com/apache/incubator-linkis/pull/1264
Because pyspark obtains the spark version number through spark submit -- version, and converts the version number into an integer. For example, spark-2.4.1 has a version number of 241, but the spark version of CDH is 2.4.1-cdh6 3.2 this type cannot be converted to integer. The spark version of HDP is 2.3.2.3.1.4.0-315. After conversion, it is 2.3.2.3.1.4.0-315, and then an error will be reported。
Therefore, I withdraw the top five digits of the version number, such as 2.4.1-cdh6 3.2, take 2.4.1 as the version number, which is similar to HDP and can meet the requirements。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For additional commands, e-mail: dev-help@linkis.apache.org
[GitHub] [incubator-linkis] Zosimer closed pull request #1264: Pyspark support CDH and HDP versions
Posted by GitBox <gi...@apache.org>.
Zosimer closed pull request #1264:
URL: https://github.com/apache/incubator-linkis/pull/1264
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For additional commands, e-mail: dev-help@linkis.apache.org
[GitHub] [incubator-linkis] peacewong commented on a change in pull request #1264: Pyspark support CDH and HDP versions
Posted by GitBox <gi...@apache.org>.
peacewong commented on a change in pull request #1264:
URL: https://github.com/apache/incubator-linkis/pull/1264#discussion_r777796628
##########
File path: linkis-engineconn-plugins/engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/executor/SparkPythonExecutor.scala
##########
@@ -161,7 +161,7 @@ class SparkPythonExecutor(val sparkEngineSession: SparkEngineSession, val id: In
val cmd = CommandLine.parse(pythonExec)
cmd.addArgument(createFakeShell(pythonScriptPath).getAbsolutePath, false)
cmd.addArgument(port.toString, false)
- cmd.addArgument(EngineUtils.sparkSubmitVersion().replaceAll("\\.", ""), false)
Review comment:
This bug can be modified by referring to the [ 0.11.0 method](https://github.com/apache/incubator-linkis/commit/0486a9abf3c2c2e1261195e41e758f5b9f0f9c78#diff-597ca2ca884fe266f156082c9c6a316435116a41c9ae56e8cbc9a56eea23fbee)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For additional commands, e-mail: dev-help@linkis.apache.org