You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/08 06:21:51 UTC
[GitHub] [spark] AngersZhuuuu opened a new pull request #33934: [SPARK-36691][PYSPARK] PythonRunner failed should pass error message to ApplicationMaster too
AngersZhuuuu opened a new pull request #33934:
URL: https://github.com/apache/spark/pull/33934
### What changes were proposed in this pull request?
In current pyspark, stderr and stdout are print together, if python script exit, PythonRunner will only throw a `SparkUserAppsException` with exit code 1. Then pass this error to AM.
In cluster mode, client side only got exception `SparkUserAppsException` and show
```
User application exited with 1.
```
Without correct error message. Then user need to check ApplicationMaster's stdout log file to find out why their job failed.
In this pr, make PythonRunner can throw exception message to backend.
### Why are the changes needed?
Make user to know error message more easy.
### Does this PR introduce _any_ user-facing change?
In cluster mode, user can directly see pyspark's error message in client side.
### How was this patch tested?
If we run a sql with wrong table in python script. In ApplicationMaster and client side log will show
```
21/09/08 14:08:42 ERROR Client: Application diagnostics message: User application exited with 1.
Exception in thread "main" org.apache.spark.SparkException: Application application_1630930053097_708441 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1150)
at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1530)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```
Now will show
```
21/09/08 14:08:42 ERROR Client: Application diagnostics message: User application exited with 1 and error message Traceback (most recent call last):
File "test.py", line 68, in <module>
res = client.sql(exec_sql)
File "/mnt/ssd/0/yarn/nm-local-dir/usercache/yi.zhu/appcache/application_1630930053097_708441/container_e236_1630930053097_708441_02_000002/pyspark.zip/pyspark/sql/session.py", line 767, in sql
File "/mnt/ssd/0/yarn/nm-local-dir/usercache/yi.zhu/appcache/application_1630930053097_708441/container_e236_1630930053097_708441_02_000002/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
File "/mnt/ssd/0/yarn/nm-local-dir/usercache/yi.zhu/appcache/application_1630930053097_708441/container_e236_1630930053097_708441_02_000002/pyspark.zip/pyspark/sql/utils.py", line 69, in deco
pyspark.sql.utils.AnalysisException: u"Table or view not found: `shopee`.`trafficixadwadwa_mart_dwd__click_di`; line 14 pos 9;\n'InsertIntoTable 'UnresolvedRelation `shopee`.`fact_shopee_bp_traffic_mart_click_di`, Map(dt -> None, country -> None), true, false\n+- 'Repartition 50, true\n +- 'Project [cast('get_json_object('data, $.shopid) as bigint) AS shopid#4, cast('get_json_object('data, $.itemid) as bigint) AS itemid#5, cast('get_json_object('data, $.quantity) as bigint) AS quantity#6, 'userid, 'platform, 'page_type, 'log_timestamp, 'utc_date AS dt#7, 'grass_region AS country#8]\n +- 'Filter ((('utc_date = cast(2021-01-01 as date)) && ('grass_region = ID)) && ('operation = action_add_to_cart_success))\n +- 'SubqueryAlias `di`\n +- 'UnresolvedRelation `shopee`.`trafficixadwadwa_mart_dwd__click_di`\n"
Exception in thread "main" org.apache.spark.SparkException: Application application_1630930053097_708441 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1150)
at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1530)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-916048630
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143118/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915080908
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143074/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYSPARK] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-914969142
**[Test build #143074 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143074/testReport)** for PR 33934 at commit [`6da39fd`](https://github.com/apache/spark/commit/6da39fd5c44f3666f2e65e9ce159d2ec324bee78).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915755721
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47608/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33934: [SPARK-36691][PYSPARK] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915029441
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-918263328
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47711/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33934: [SPARK-36691][PYSPARK] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915029441
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-918234102
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47711/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33934:
URL: https://github.com/apache/spark/pull/33934#discussion_r707310255
##########
File path: core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala
##########
@@ -96,11 +96,13 @@ object PythonRunner {
try {
val process = builder.start()
- new RedirectThread(process.getInputStream, System.out, "redirect output").start()
+ val redirectOutput = new RedirectThreadAndCatchErrorMsg(
+ process.getInputStream, System.out, "redirect output", "Traceback")
Review comment:
> I think we should
>
> 1. remove `builder.redirectErrorStream`
> 2. find an easy way to merge both stderr & stdout to print out later (manually on JVM)
> 3. extract stderr from `process`
> 4. use it in exception message
Since traceback is print before Process exit, so we still need to extract error message from output. Or we just use all information in stderr as exception message?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915082819
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143069/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-921737048
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47918/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33934:
URL: https://github.com/apache/spark/pull/33934#discussion_r710761839
##########
File path: core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala
##########
@@ -96,11 +96,13 @@ object PythonRunner {
try {
val process = builder.start()
- new RedirectThread(process.getInputStream, System.out, "redirect output").start()
+ val redirectOutput = new RedirectThreadAndCatchErrorMsg(
+ process.getInputStream, System.out, "redirect output", "Traceback")
Review comment:
yeah, i thought to show the stderr as error message. I don't think there's a claver way to extract Python exception cleanly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-916034213
**[Test build #143118 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143118/testReport)** for PR 33934 at commit [`c905b7b`](https://github.com/apache/spark/commit/c905b7bb3d0a85187e4348dcfc59ba3f347632ff).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915080908
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143074/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915081053
**[Test build #143069 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143069/testReport)** for PR 33934 at commit [`5ade9f0`](https://github.com/apache/spark/commit/5ade9f0649b3c20d09f6f3e8737588f3616ac8d6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915906130
**[Test build #143110 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143110/testReport)** for PR 33934 at commit [`1a017c7`](https://github.com/apache/spark/commit/1a017c7879d45cb1caaf3f49e3f4ce26c76116bb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-921813431
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143410/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-921668941
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47914/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915965098
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47622/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915921354
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143110/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-921809491
**[Test build #143410 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143410/testReport)** for PR 33934 at commit [`8058f28`](https://github.com/apache/spark/commit/8058f28c46b122d023b09857ffaec55f02ba09b6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915925185
**[Test build #143118 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143118/testReport)** for PR 33934 at commit [`c905b7b`](https://github.com/apache/spark/commit/c905b7bb3d0a85187e4348dcfc59ba3f347632ff).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-917409079
**[Test build #143166 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143166/testReport)** for PR 33934 at commit [`7968f99`](https://github.com/apache/spark/commit/7968f99a75fb2dc114e19ef13d8167a373e1bf6d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-918343435
**[Test build #143208 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143208/testReport)** for PR 33934 at commit [`1fc50d7`](https://github.com/apache/spark/commit/1fc50d778c96521a8ecd6b57c5e4360714582e53).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYSPARK] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-914983653
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47571/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915735946
**[Test build #143104 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143104/testReport)** for PR 33934 at commit [`1a017c7`](https://github.com/apache/spark/commit/1a017c7879d45cb1caaf3f49e3f4ce26c76116bb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33934: [SPARK-36691][PYSPARK] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33934:
URL: https://github.com/apache/spark/pull/33934#discussion_r704085997
##########
File path: core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala
##########
@@ -92,15 +92,19 @@ object PythonRunner {
// see https://github.com/numpy/numpy/issues/10455
sparkConf.getOption("spark.driver.cores").foreach(env.put("OMP_NUM_THREADS", _))
}
- builder.redirectErrorStream(true) // Ugly but needed for stdout and stderr to synchronize
try {
val process = builder.start()
+ val stderrBuffer = new CircularBuffer(2048)
+ new RedirectThread(
+ process.getErrorStream,
+ stderrBuffer,
+ "Thread-ScriptTransformation-STDERR-Consumer").start()
Review comment:
Why is it ScriptTransform?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915778293
**[Test build #143104 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143104/testReport)** for PR 33934 at commit [`1a017c7`](https://github.com/apache/spark/commit/1a017c7879d45cb1caaf3f49e3f4ce26c76116bb).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-921704446
**[Test build #143410 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143410/testReport)** for PR 33934 at commit [`8058f28`](https://github.com/apache/spark/commit/8058f28c46b122d023b09857ffaec55f02ba09b6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915836280
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47613/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915082819
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143069/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-917435386
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143166/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915803919
**[Test build #143110 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143110/testReport)** for PR 33934 at commit [`1a017c7`](https://github.com/apache/spark/commit/1a017c7879d45cb1caaf3f49e3f4ce26c76116bb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-918345899
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143208/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-1007523767
@holdenk who may have dealt with the python side before and have thoughts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915778870
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143104/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33934: [SPARK-36691][PYSPARK] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33934:
URL: https://github.com/apache/spark/pull/33934#discussion_r704085749
##########
File path: core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala
##########
@@ -92,15 +92,19 @@ object PythonRunner {
// see https://github.com/numpy/numpy/issues/10455
sparkConf.getOption("spark.driver.cores").foreach(env.put("OMP_NUM_THREADS", _))
}
- builder.redirectErrorStream(true) // Ugly but needed for stdout and stderr to synchronize
try {
val process = builder.start()
+ val stderrBuffer = new CircularBuffer(2048)
+ new RedirectThread(
+ process.getErrorStream,
+ stderrBuffer,
+ "Thread-ScriptTransformation-STDERR-Consumer").start()
new RedirectThread(process.getInputStream, System.out, "redirect output").start()
Review comment:
Looks like now it won't show the stderr in stdout anymore?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-921813431
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143410/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-918263328
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47711/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-921729929
**[Test build #143406 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143406/testReport)** for PR 33934 at commit [`421370b`](https://github.com/apache/spark/commit/421370bf2373ebddd0dc9e259f1fc473a115201b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915079395
**[Test build #143074 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143074/testReport)** for PR 33934 at commit [`6da39fd`](https://github.com/apache/spark/commit/6da39fd5c44f3666f2e65e9ce159d2ec324bee78).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33934:
URL: https://github.com/apache/spark/pull/33934#discussion_r704875880
##########
File path: core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala
##########
@@ -92,15 +92,19 @@ object PythonRunner {
// see https://github.com/numpy/numpy/issues/10455
sparkConf.getOption("spark.driver.cores").foreach(env.put("OMP_NUM_THREADS", _))
}
- builder.redirectErrorStream(true) // Ugly but needed for stdout and stderr to synchronize
try {
val process = builder.start()
+ val stderrBuffer = new CircularBuffer(2048)
+ new RedirectThread(
+ process.getErrorStream,
+ stderrBuffer,
+ "Thread-ScriptTransformation-STDERR-Consumer").start()
new RedirectThread(process.getInputStream, System.out, "redirect output").start()
Review comment:
Is there any easy way to get the stderr stream, and manually merge stderr and stdout to printout? I think we shouldn't remove this from stdout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYSPARK] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915002768
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47577/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYSPARK] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915008792
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47571/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915925185
**[Test build #143118 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143118/testReport)** for PR 33934 at commit [`c905b7b`](https://github.com/apache/spark/commit/c905b7bb3d0a85187e4348dcfc59ba3f347632ff).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915959219
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47622/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915778870
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143104/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-921735137
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143406/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-921662508
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47914/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33934:
URL: https://github.com/apache/spark/pull/33934#discussion_r704925379
##########
File path: core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala
##########
@@ -92,15 +92,19 @@ object PythonRunner {
// see https://github.com/numpy/numpy/issues/10455
sparkConf.getOption("spark.driver.cores").foreach(env.put("OMP_NUM_THREADS", _))
}
- builder.redirectErrorStream(true) // Ugly but needed for stdout and stderr to synchronize
try {
val process = builder.start()
+ val stderrBuffer = new CircularBuffer(2048)
+ new RedirectThread(
+ process.getErrorStream,
+ stderrBuffer,
+ "Thread-ScriptTransformation-STDERR-Consumer").start()
new RedirectThread(process.getInputStream, System.out, "redirect output").start()
Review comment:
> Is there any easy way to get the stderr stream, and manually merge stderr and stdout to printout? I think we shouldn't remove this from stdout.
How about current?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33934:
URL: https://github.com/apache/spark/pull/33934#discussion_r710998960
##########
File path: core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala
##########
@@ -96,11 +96,13 @@ object PythonRunner {
try {
val process = builder.start()
- new RedirectThread(process.getInputStream, System.out, "redirect output").start()
+ val redirectOutput = new RedirectThreadAndCatchErrorMsg(
+ process.getInputStream, System.out, "redirect output", "Traceback")
Review comment:
> yeah, i thought to show the stderr as error message. I don't think there's a claver way to extract Python exception cleanly.
In current code can satisfy this requirement, but can't keep stdout and stderr all sync.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-1007523767
@holdenk who may have dealt with the python side before and have thoughts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915836086
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47613/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-921735137
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143406/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-921732611
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47918/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-921704446
**[Test build #143410 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143410/testReport)** for PR 33934 at commit [`8058f28`](https://github.com/apache/spark/commit/8058f28c46b122d023b09857ffaec55f02ba09b6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-918164135
**[Test build #143208 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143208/testReport)** for PR 33934 at commit [`1fc50d7`](https://github.com/apache/spark/commit/1fc50d778c96521a8ecd6b57c5e4360714582e53).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915758874
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47608/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33934:
URL: https://github.com/apache/spark/pull/33934#discussion_r706730455
##########
File path: core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala
##########
@@ -96,11 +96,13 @@ object PythonRunner {
try {
val process = builder.start()
- new RedirectThread(process.getInputStream, System.out, "redirect output").start()
+ val redirectOutput = new RedirectThreadAndCatchErrorMsg(
+ process.getInputStream, System.out, "redirect output", "Traceback")
Review comment:
I think we should
1. remove `builder.redirectErrorStream`
2. find an easy way to merge both stderr & stdout to print out later (manually on JVM)
3. extract stderr from `process`
4. use it in exception message
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-917417678
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47670/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915965064
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47622/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYSPARK] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-914957902
**[Test build #143069 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143069/testReport)** for PR 33934 at commit [`5ade9f0`](https://github.com/apache/spark/commit/5ade9f0649b3c20d09f6f3e8737588f3616ac8d6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-914957902
**[Test build #143069 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143069/testReport)** for PR 33934 at commit [`5ade9f0`](https://github.com/apache/spark/commit/5ade9f0649b3c20d09f6f3e8737588f3616ac8d6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-1048390404
gentle ping @holdenk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33934: [SPARK-36691][PYSPARK] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-914953368
ping @HyukjinKwon @cloud-fan @dongjoon-hyun @vanzin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-975041128
gentle ping @tgravescs and @mridulm, this issue make many pyspark dev complain this.
But I didn't find the way to realize what @HyukjinKwon suggested in https://github.com/apache/spark/pull/33934#discussion_r706730455
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915758874
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47608/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYSPARK] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915010404
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47577/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-916727089
ping @HyukjinKwon @yaooqinn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915735946
**[Test build #143104 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143104/testReport)** for PR 33934 at commit [`1a017c7`](https://github.com/apache/spark/commit/1a017c7879d45cb1caaf3f49e3f4ce26c76116bb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33934:
URL: https://github.com/apache/spark/pull/33934#discussion_r708043794
##########
File path: core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala
##########
@@ -96,11 +96,13 @@ object PythonRunner {
try {
val process = builder.start()
- new RedirectThread(process.getInputStream, System.out, "redirect output").start()
+ val redirectOutput = new RedirectThreadAndCatchErrorMsg(
+ process.getInputStream, System.out, "redirect output", "Traceback")
Review comment:
gentle ping @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33934:
URL: https://github.com/apache/spark/pull/33934#discussion_r707225453
##########
File path: core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala
##########
@@ -96,11 +96,13 @@ object PythonRunner {
try {
val process = builder.start()
- new RedirectThread(process.getInputStream, System.out, "redirect output").start()
+ val redirectOutput = new RedirectThreadAndCatchErrorMsg(
+ process.getInputStream, System.out, "redirect output", "Traceback")
Review comment:
> I think we should
>
> 1. remove `builder.redirectErrorStream`
> 2. find an easy way to merge both stderr & stdout to print out later (manually on JVM)
> 3. extract stderr from `process`
> 4. use it in exception message
You mean the whole stderr as exception message?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915803919
**[Test build #143110 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143110/testReport)** for PR 33934 at commit [`1a017c7`](https://github.com/apache/spark/commit/1a017c7879d45cb1caaf3f49e3f4ce26c76116bb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-917536259
@tgravescs and @mridulm too FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-917416045
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47670/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915790279
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-918345899
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143208/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-918164135
**[Test build #143208 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143208/testReport)** for PR 33934 at commit [`1fc50d7`](https://github.com/apache/spark/commit/1fc50d778c96521a8ecd6b57c5e4360714582e53).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-1007946564
> @holdenk who may have dealt with the python side before and have thoughts
Thanks for your ping. It's a really useful feature and my wired pr have be deployed to our prod. Make user convenient to check failed reason.
Hope some suggestion to make this feature to be continued.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915921354
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143110/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-917417239
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47670/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33934: [SPARK-36691][PYSPARK] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33934:
URL: https://github.com/apache/spark/pull/33934#discussion_r704107765
##########
File path: core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala
##########
@@ -92,15 +92,19 @@ object PythonRunner {
// see https://github.com/numpy/numpy/issues/10455
sparkConf.getOption("spark.driver.cores").foreach(env.put("OMP_NUM_THREADS", _))
}
- builder.redirectErrorStream(true) // Ugly but needed for stdout and stderr to synchronize
try {
val process = builder.start()
+ val stderrBuffer = new CircularBuffer(2048)
+ new RedirectThread(
+ process.getErrorStream,
+ stderrBuffer,
+ "Thread-ScriptTransformation-STDERR-Consumer").start()
new RedirectThread(process.getInputStream, System.out, "redirect output").start()
Review comment:
> Looks like now it won't show the stderr in stdout anymore?
It's a problem. If we reserve `builder.redirectErrorStream(true)`
We need to split message in inputStream and got error message after `Traceback` as error message?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-914969142
**[Test build #143074 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143074/testReport)** for PR 33934 at commit [`6da39fd`](https://github.com/apache/spark/commit/6da39fd5c44f3666f2e65e9ce159d2ec324bee78).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-917435386
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143166/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33934:
URL: https://github.com/apache/spark/pull/33934#discussion_r710998960
##########
File path: core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala
##########
@@ -96,11 +96,13 @@ object PythonRunner {
try {
val process = builder.start()
- new RedirectThread(process.getInputStream, System.out, "redirect output").start()
+ val redirectOutput = new RedirectThreadAndCatchErrorMsg(
+ process.getInputStream, System.out, "redirect output", "Traceback")
Review comment:
> yeah, i thought to show the stderr as error message. I don't think there's a claver way to extract Python exception cleanly.
In current code can satisfy this requirement, but can't keep stdout and stderr all sync. Do you have any suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-917409079
**[Test build #143166 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143166/testReport)** for PR 33934 at commit [`7968f99`](https://github.com/apache/spark/commit/7968f99a75fb2dc114e19ef13d8167a373e1bf6d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915830617
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47613/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-921626507
**[Test build #143406 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143406/testReport)** for PR 33934 at commit [`421370b`](https://github.com/apache/spark/commit/421370bf2373ebddd0dc9e259f1fc473a115201b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-921737073
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47918/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33934: [SPARK-36691][PYSPARK] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33934:
URL: https://github.com/apache/spark/pull/33934#discussion_r704101357
##########
File path: core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala
##########
@@ -92,15 +92,19 @@ object PythonRunner {
// see https://github.com/numpy/numpy/issues/10455
sparkConf.getOption("spark.driver.cores").foreach(env.put("OMP_NUM_THREADS", _))
}
- builder.redirectErrorStream(true) // Ugly but needed for stdout and stderr to synchronize
try {
val process = builder.start()
+ val stderrBuffer = new CircularBuffer(2048)
+ new RedirectThread(
+ process.getErrorStream,
+ stderrBuffer,
+ "Thread-ScriptTransformation-STDERR-Consumer").start()
Review comment:
> Why is it ScriptTransform?
Updated to PythonRunner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-921668959
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47914/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-921737073
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47918/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-921626507
**[Test build #143406 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143406/testReport)** for PR 33934 at commit [`421370b`](https://github.com/apache/spark/commit/421370bf2373ebddd0dc9e259f1fc473a115201b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33934:
URL: https://github.com/apache/spark/pull/33934#discussion_r706730455
##########
File path: core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala
##########
@@ -96,11 +96,13 @@ object PythonRunner {
try {
val process = builder.start()
- new RedirectThread(process.getInputStream, System.out, "redirect output").start()
+ val redirectOutput = new RedirectThreadAndCatchErrorMsg(
+ process.getInputStream, System.out, "redirect output", "Traceback")
Review comment:
I think we should
1. remove `builder.redirectErrorStream`
2. find an easy way to merge both stderr & stdout to print out later
3. extract stderr from `process`
4. use it in exception message
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915965098
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47622/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-916048630
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143118/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-917431740
**[Test build #143166 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143166/testReport)** for PR 33934 at commit [`7968f99`](https://github.com/apache/spark/commit/7968f99a75fb2dc114e19ef13d8167a373e1bf6d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915758856
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47608/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-917417678
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47670/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-918245502
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47711/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-915836280
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47613/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33934: [SPARK-36691][PYTHON] PythonRunner failed should pass error message to ApplicationMaster too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33934:
URL: https://github.com/apache/spark/pull/33934#issuecomment-921668959
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47914/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org