You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "j1wonpark (via GitHub)" <gi...@apache.org> on 2026/08/21 13:43:25 UTC
[PR] [SPARK-58932][SS] Close the accepted socket in TransformWithStateInPySparkStateServer [spark]
j1wonpark opened a new pull request, #58205:
URL: https://github.com/apache/spark/pull/58205
### What changes were proposed in this pull request?
Run the request loop of `TransformWithStateInPySparkStateServer.run()` inside `Utils.tryWithResource` so the socket returned by `stateServerSocket.accept()` is always closed.
### Why are the changes needed?
`run()` accepts a connection from the Python worker but never closes it. The task completion listener closes only the listening `ServerSocketChannel`, so the accepted connection's file descriptor is held until the executor exits.
`initStateServer()` binds an ephemeral port per task, so a long-running streaming query leaks one socket per task. Once the ephemeral port range is exhausted, every new connection on that executor fails with `java.net.BindException: Cannot assign requested address`, which also takes down unrelated connections such as the Kafka source.
The loop has several early returns, so the close has to be tied to the scope.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Two tests added to `TransformWithStateInPySparkStateServerSuite`, covering the path where the request loop is never entered and the EOF early-return path. Both fail without the fix. The suite previously had no test calling `run()`.
### Was this patch authored or co-authored using generative AI tooling?
Yes. Generated-by: Claude Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58932][SS] Close the accepted socket in TransformWithStateInPySparkStateServer [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #58205:
URL: https://github.com/apache/spark/pull/58205#discussion_r3860667123
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:
##########
@@ -157,6 +159,13 @@ class TransformWithStateInPySparkStateServer(
return
}
+ // The task completion listener closes only the listening server socket, and the
+ // request loop has several early returns, so the accepted connection is closed
+ // through tryWithResource.
+ Utils.tryWithResource(listeningSocket)(serveRequests)
+ }
+
+ private def serveRequests(listeningSocket: SocketChannel): Unit = {
Review Comment:
Neat approach to avoid indentation :) Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58932][SS] Close the accepted socket in TransformWithStateInPySparkStateServer [spark]
Posted by "j1wonpark (via GitHub)" <gi...@apache.org>.
j1wonpark commented on PR #58205:
URL: https://github.com/apache/spark/pull/58205#issuecomment-5420909349
@HeartSaVioR Thanks! Merged the latest master, keeping the exception handling from SPARK-58977 while closing the accepted socket via `Utils.tryWithResource`. All tests in the suite pass locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58932][SS] Close the accepted socket in TransformWithStateInPySparkStateServer [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #58205:
URL: https://github.com/apache/spark/pull/58205#issuecomment-5422281681
@j1wonpark
I've merged the fix till 4.3 which we hope to release in a month. Please submit a PR for older version branch if you were aiming to have the fix for older version lines. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58932][SS] Close the accepted socket in TransformWithStateInPySparkStateServer [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #58205:
URL: https://github.com/apache/spark/pull/58205#issuecomment-5420501107
@j1wonpark
Thanks for the contribution!
I see we missed to close the socket, but we still need to take care of some exception cases, hence the merge conflict. Could you please rebase and retain the exception handling while ensuring the closure of the socket?
Thanks in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58932][SS] Close the accepted socket in TransformWithStateInPySparkStateServer [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #58205:
URL: https://github.com/apache/spark/pull/58205#issuecomment-5371267092
Thank you @j1wonpark!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58932][SS] Close the accepted socket in TransformWithStateInPySparkStateServer [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #58205:
URL: https://github.com/apache/spark/pull/58205#issuecomment-5422260095
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/c5fbdf74b6886bd2c21a5cd4bc5c869e490abe63
- merged into branch-4.x https://github.com/apache/spark/commit/516942feab19138e7708b5f554106bece575f278
- merged into branch-4.3 https://github.com/apache/spark/commit/818c2c8614321a670aed23d6374b7bfa6052e280
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58932][SS] Close the accepted socket in TransformWithStateInPySparkStateServer [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR closed pull request #58205: [SPARK-58932][SS] Close the accepted socket in TransformWithStateInPySparkStateServer
URL: https://github.com/apache/spark/pull/58205
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58932][SS] Close the accepted socket in TransformWithStateInPySparkStateServer [spark]
Posted by "j1wonpark (via GitHub)" <gi...@apache.org>.
j1wonpark commented on PR #58205:
URL: https://github.com/apache/spark/pull/58205#issuecomment-5423781855
@HeartSaVioR Opened backport PRs for the older version lines: #58308 (branch-4.2) and #58309 (branch-4.1). Both branches lack SPARK-58977, so the cherry-pick keeps `accept()` unchanged and only wraps the request loop. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org