You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2024/01/08 00:01:53 UTC
[PR] [SPARK-46613][SQL][PYTHON] Log full exception when failed to lookup Python Data Sources [spark]
HyukjinKwon opened a new pull request, #44617:
URL: https://github.com/apache/spark/pull/44617
### What changes were proposed in this pull request?
This PR proposes to log full exception when failed to lookup Python Data Sources
### Why are the changes needed?
In my internal testing it logs something like:
```
...
24/01/05 03:49:49 WARN DataSourceManager: Skipping the lookup of Python Data Sources due to the failure: java.lang.StackOverflowError
24/01/05 03:49:49 WARN DataSourceManager: Skipping the lookup of Python Data Sources due to the failure: java.lang.StackOverflowError
24/01/05 03:49:49 WARN WSFSWorkerManager: Files feature is disabled because com.databricks.backend.daemon.driver.WSFSManager is not available
24/01/05 03:49:49 WARN DataSourceManager: Skipping the lookup of Python Data Sources due to the failure: java.lang.StackOverflowError
24/01/05 03:49:49 WARN DataSourceManager: Skipping the lookup of Python Data Sources due to the failure: java.lang.StackOverflowError
24/01/05 03:49:49 WARN PythonWorkerFactory: Failed to open socket to Python daemon:
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
...
```
which is hard to debug. It should show the full error messages so developers can debug.
### Does this PR introduce _any_ user-facing change?
No, the main change has not been released out yet.
### How was this patch tested?
Manually.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46613][SQL][PYTHON] Log full exception when failed to lookup Python Data Sources [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #44617:
URL: https://github.com/apache/spark/pull/44617#issuecomment-1880233908
Build https://github.com/HyukjinKwon/spark/actions/runs/7441506339
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46613][SQL][PYTHON] Log full exception when failed to lookup Python Data Sources [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #44617: [SPARK-46613][SQL][PYTHON] Log full exception when failed to lookup Python Data Sources
URL: https://github.com/apache/spark/pull/44617
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46613][SQL][PYTHON] Log full exception when failed to lookup Python Data Sources [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #44617:
URL: https://github.com/apache/spark/pull/44617#issuecomment-1880262341
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org