You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "seilylook (via GitHub)" <gi...@apache.org> on 2024/12/18 02:39:10 UTC
[PR] [SPARK-40993][CONNECT][PYTHON][DOCS] Migrate markdown style README to PySpark Development Documentation [spark]
seilylook opened a new pull request, #49221:
URL: https://github.com/apache/spark/pull/49221
### What changes were proposed in this pull request?
Added troubleshooting instructions for Spark driver binding issues, including setting `SPARK_LOCAL_IP` to `127.0.0.1` and running Spark shell as a workaround.
### Why are the changes needed?
Users may encounter an issue where the Spark driver cannot bind to a random free port, leading to errors during initialization. Providing clear instructions will help users quickly resolve this problem.
### Does this PR introduce any user-facing change?
Yes, it adds troubleshooting steps to address Spark driver binding issues.
### How was this patch tested?
Manually verified by setting `SPARK_LOCAL_IP` to `127.0.0.1` and running Spark shell. Confirmed that the issue no longer occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-40993][CONNECT][PYTHON][DOCS] Migrate markdown style README to PySpark Development Documentation [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49221:
URL: https://github.com/apache/spark/pull/49221#issuecomment-2764310154
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-40993][CONNECT][PYTHON][DOCS] Migrate markdown style README to PySpark Development Documentation [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #49221:
URL: https://github.com/apache/spark/pull/49221#discussion_r1889877585
##########
python/docs/source/development/testing.rst:
##########
@@ -38,6 +38,27 @@ After that, the PySpark test cases can be run via using ``python/run-tests``. Fo
Note that you may set ``OBJC_DISABLE_INITIALIZE_FORK_SAFETY`` environment variable to ``YES`` if you are running tests on Mac OS.
+.. note::
+
+ If the Spark driver is unavailable, you can resolve the issue using the following methods:
+
+ **Set SPARK_LOCAL_IP**:
+
+ Configure the environment variable ``SPARK_LOCAL_IP`` to bind to the local address ``127.0.0.1``::
+
+ export SPARK_LOCAL_IP=127.0.0.1
Review Comment:
I actually don't think this is specific to PySpark but Spark development in general. Since we don't have a page for Spark itself yet, maybe we can add the details into https://spark.apache.org/docs/latest/api/python/development/debugging.html#common-exceptions-errors for now.
It'd be great if we can explain the background and details about why it has to be set, and why it happens.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-40993][CONNECT][PYTHON][DOCS] Migrate markdown style README to PySpark Development Documentation [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #49221: [SPARK-40993][CONNECT][PYTHON][DOCS] Migrate markdown style README to PySpark Development Documentation
URL: https://github.com/apache/spark/pull/49221
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-40993][CONNECT][PYTHON][DOCS] Migrate markdown style README to PySpark Development Documentation [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #49221:
URL: https://github.com/apache/spark/pull/49221#issuecomment-2550765646
Let's also file a new JIRA for this, see also https://spark.apache.org/contributing.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-40993][CONNECT][PYTHON][DOCS] Migrate markdown style README to PySpark Development Documentation [spark]
Posted by "allisonwang-db (via GitHub)" <gi...@apache.org>.
allisonwang-db commented on code in PR #49221:
URL: https://github.com/apache/spark/pull/49221#discussion_r1891551660
##########
python/docs/source/development/testing.rst:
##########
@@ -38,6 +38,27 @@ After that, the PySpark test cases can be run via using ``python/run-tests``. Fo
Note that you may set ``OBJC_DISABLE_INITIALIZE_FORK_SAFETY`` environment variable to ``YES`` if you are running tests on Mac OS.
+.. note::
+
+ If the Spark driver is unavailable, you can resolve the issue using the following methods:
+
+ **Set SPARK_LOCAL_IP**:
+
+ Configure the environment variable ``SPARK_LOCAL_IP`` to bind to the local address ``127.0.0.1``::
+
+ export SPARK_LOCAL_IP=127.0.0.1
Review Comment:
Yea adding this to debugging.rst would be better, and we have a reference link mentioned in the testing guide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org