You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "TomeHirata (via GitHub)" <gi...@apache.org> on 2026/08/28 06:19:59 UTC
[PR] [SPARK-59070][PYTHON][ML] Guard TorchDistributor log socket getpeername() against OSError [spark]
TomeHirata opened a new pull request, #58375:
URL: https://github.com/apache/spark/pull/58375
### What changes were proposed in this pull request?
In `TorchDistributor._execute_command`, the loop that streams subprocess output compares `log_streaming_client.sock.getsockname()[0]` with `log_streaming_client.sock.getpeername()[0]` to decide whether to echo each log line to stdout (avoiding duplication when client and server run on the same node). This comparison is now wrapped in a `try/except OSError` block. If the socket call raises, `same_node` falls back to `False`, and the line is written to stdout.
### Why are the changes needed?
The log-streaming socket is best effort and can be dropped by a cloud-provider idle-timeout while torch/NCCL initializes (roughly 24 seconds of silence). The next log line then calls `getpeername()` on the dead socket, raising `OSError: [Errno 107] Transport endpoint is not connected`. Because `_execute_command` runs inside a barrier task, this uncaught error aborts the entire multi-node training job approximately 30 seconds in.
`LogStreamingClient.send()` in `log_communication.py` already guards against this case (try/except → mark client failed); the getsockname/getpeername comparison was the only remaining unguarded call site.
Reproduced on Databricks Runtime 17.3.19 ML GPU against Apache Spark 4.0.
### Does this PR introduce _any_ user-facing change?
No API change. Multi-node `TorchDistributor` training that previously crashed with `OSError: [Errno 107] Transport endpoint is not connected` ~30 s into a run will now degrade the best-effort log socket gracefully and continue training.
### How was this patch tested?
Added `TorchDistributorBaselineUnitTests.test_execute_command_survives_log_socket_drop` in `python/pyspark/ml/torch/tests/test_distributor.py`. The test configures a mock `LogStreamingClient` whose `getpeername()` raises `OSError(107, ...)`, then asserts that `_execute_command` still completes and the output line reaches stdout.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude (claude-sonnet-4-6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59070][PYTHON][ML] Guard TorchDistributor log socket getpeername() against OSError [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #58375:
URL: https://github.com/apache/spark/pull/58375#issuecomment-5449933021
@TomeHirata thanks for the fix.
it seems you need to enable the CI in your fork
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59070][PYTHON][ML] Guard TorchDistributor log socket getpeername() against OSError [spark]
Posted by "TomeHirata (via GitHub)" <gi...@apache.org>.
TomeHirata commented on PR #58375:
URL: https://github.com/apache/spark/pull/58375#issuecomment-5507833582
@zhengruifeng thanks! enabled the CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59070][PYTHON][ML] Guard TorchDistributor log socket getpeername() against OSError [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #58375:
URL: https://github.com/apache/spark/pull/58375#issuecomment-5511092250
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/b519a3fdf48d2889817e15f0a2aeb34e59f0a923
- merged into branch-4.x https://github.com/apache/spark/commit/c1ef95b09832aa81acbd45fdfa4fee5c3f451ecc
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59070][PYTHON][ML] Guard TorchDistributor log socket getpeername() against OSError [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #58375: [SPARK-59070][PYTHON][ML] Guard TorchDistributor log socket getpeername() against OSError
URL: https://github.com/apache/spark/pull/58375
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org