You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@linkis.apache.org by "casionone (via GitHub)" <gi...@apache.org> on 2023/03/05 11:22:34 UTC
[GitHub] [linkis] casionone opened a new pull request, #4295: fix python watch dog not work
casionone opened a new pull request, #4295:
URL: https://github.com/apache/linkis/pull/4295
<!--
Thank you for sending the PR! We appreciate you spending the time to work on these changes.
You can learn more about contributing to Apache Linkis here: https://linkis.apache.org/community/how-to-contribute
Happy contributing!
-->
### Related issues/PRs
Related issues: #4294
### Brief change log
optimize pyspark watch dog
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] peacewong commented on a diff in pull request #4295: fix python watch dog not work
Posted by "peacewong (via GitHub)" <gi...@apache.org>.
peacewong commented on code in PR #4295:
URL: https://github.com/apache/linkis/pull/4295#discussion_r1126350065
##########
linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/executor/SparkPythonExecutor.scala:
##########
@@ -187,11 +194,24 @@ class SparkPythonExecutor(val sparkEngineSession: SparkEngineSession, val id: In
.foreach(pythonClasspath ++= File.pathSeparator ++= _)
val cmd = CommandLine.parse(pythonExec)
- cmd.addArgument(createFakeShell(pythonScriptPath).getAbsolutePath, false)
+ if (SparkConfiguration.SPARK_PYTHON_TEST_MODE_ENABLE.getValue) {
+ val path = SparkConfiguration.SPARK_PYTHON_TEST_MODE_MIX__PYSHELL_PATH.getValue
+ logger.info(
+ s"wds.linkis.spark.python.test.mode.enable => true, will use wds.linkis.spark.python.mix.pyshell.path:${path}"
Review Comment:
Should SparkConfiguration.SPARK_PYTHON_TEST_MODE_ENABLE.key
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] peacewong commented on pull request #4295: fix python watch dog not work
Posted by "peacewong (via GitHub)" <gi...@apache.org>.
peacewong commented on PR #4295:
URL: https://github.com/apache/linkis/pull/4295#issuecomment-1456049403
python ec need to update sys.exit to os.exit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] peacewong merged pull request #4295: fix python watch dog not work
Posted by "peacewong (via GitHub)" <gi...@apache.org>.
peacewong merged PR #4295:
URL: https://github.com/apache/linkis/pull/4295
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] jackxu2011 commented on a diff in pull request #4295: fix python watch dog not work
Posted by "jackxu2011 (via GitHub)" <gi...@apache.org>.
jackxu2011 commented on code in PR #4295:
URL: https://github.com/apache/linkis/pull/4295#discussion_r1125840177
##########
linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/executor/SparkPythonExecutor.scala:
##########
@@ -126,7 +126,7 @@ class SparkPythonExecutor(val sparkEngineSession: SparkEngineSession, val id: In
// process.destroy will kills the subprocess,not need to force kill with -9,
// kill -9 may cause resources not to be released
pid.foreach(p => {
- logger.info(s"skip Try to kill Pyspark process with: [kill -9 ${p}]")
+ logger.info(s"skip Try to kill Pyspark process with: [kill -15 ${p}]")
Utils.exec(Array("kill", "-9", p), 3000L)
Review Comment:
the log is kill -15, but the exec cmd is kill -9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] peacewong commented on a diff in pull request #4295: fix python watch dog not work
Posted by "peacewong (via GitHub)" <gi...@apache.org>.
peacewong commented on code in PR #4295:
URL: https://github.com/apache/linkis/pull/4295#discussion_r1126349081
##########
linkis-engineconn-plugins/spark/src/main/resources/python/mix_pyspark.py:
##########
@@ -209,7 +209,7 @@ def saveDFToCsv(df, path, hasheader=True,isOverwrite=False,option={}):
##add pyfiles
try:
- pyfile = sys.argv[5]
+ pyfile = sys.argv[6]
Review Comment:
should args is 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org