You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2026/06/22 19:07:04 UTC
[PR] [SPARK-57615][CONNECT][TESTS] Wait for the Connect server before creating a ResourceProfile in test_profile_before_sc_for_connect [spark]
HyukjinKwon opened a new pull request, #56676:
URL: https://github.com/apache/spark/pull/56676
### What changes were proposed in this pull request?
`test_profile_before_sc_for_connect` creates a `ResourceProfile` over Spark Connect immediately
after `SparkSession.builder.remote(...).getOrCreate()`. This PR makes the test wait for the Connect
server to be ready before doing so, using the existing `pyspark.testing.eventually` helper to retry a
trivial job until it succeeds:
```python
from pyspark.testing.utils import eventually
def _server_ready() -> bool:
spark.range(1).count()
return True
eventually(timeout=120, expected_exceptions=(Exception,))(_server_ready)()
rp.id
```
### Why are the changes needed?
The scheduled "Build / Python-only, Connect-only (Python 3.11)" build runs this test in its
`Run tests (local-cluster)` step, where the server is started with
`start-connect-server.sh --master "local-cluster[2, 4, 1024]"`. That script returns before the
local-cluster `SparkContext` is fully initialized, so the first command(s) issued against it can
fail server-side. `test_connect_resources` is the first test in that step, so it races server
startup and fails intermittently (~60% of runs), observed as a bare `java.lang.AssertionError` on
`rp.id`, or `SparkConnectGrpcException: Application error processing RPC` on the first job. When the
cluster happens to be ready, the test passes (~22-77s). Waiting for readiness first makes it
deterministic.
This is a test-only stabilization. The underlying server behavior (an internal error leaking on a
very-early command before the context is ready) is a separate, deeper robustness concern and is not
addressed here.
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
Ran the scheduled workflow (`build_python_connect.yml`) on a fork. The Connect-only build is green
end-to-end, including the previously-flaky `local-cluster` step, on two consecutive runs:
- https://github.com/HyukjinKwon/spark/actions/runs/27969109059 (attempt 1 and attempt 2: both
`Run tests (local)` and `Run tests (local-cluster)` green)
The default `build_and_test` on this branch is also green:
https://github.com/HyukjinKwon/spark/actions/runs/27973120689
Note: the Connect-only build's `Run tests (local)` step also requires the import fix in #56644
(SPARK-57598); the validation runs above were performed on a branch carrying both changes so the
`local-cluster` step is reached. This PR contains only the `test_connect_resources.py` change.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57615][CONNECT][TESTS] Wait for the Connect server before creating a ResourceProfile in test_profile_before_sc_for_connect [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #56676:
URL: https://github.com/apache/spark/pull/56676#discussion_r3455713211
##########
python/pyspark/resource/tests/test_connect_resources.py:
##########
@@ -46,6 +46,17 @@ def test_profile_before_sc_for_connect(self):
).getOrCreate()
# Still can access taskResources, similar to executorResources.
self.assertEqual(rp.taskResources["cpus"].amount, 2.0)
+ # SPARK-57615: the local-cluster Spark Connect server can return errors on the first
+ # commands right after start-connect-server.sh returns, before its SparkContext is fully
+ # initialized. Wait until a trivial job succeeds before creating a ResourceProfile, which
+ # would otherwise race that initialization and fail server-side.
+ from pyspark.testing.utils import eventually
+
+ def _server_ready() -> bool:
+ spark.range(1).count()
+ return True
+
+ eventually(timeout=120, expected_exceptions=(Exception,))(_server_ready)()
Review Comment:
Good point - narrowed it to `PySparkException` (the base class of the `SparkConnect*` exceptions the not-yet-ready server raises) instead of bare `Exception`.
I kept it at the `PySparkException` level rather than a specific `SparkConnect*` subclass for two reasons: (1) importing the connect-specific exception at module top-level would require grpc and break test collection on grpc-less builds, and (2) the not-ready server can surface the race as a few different `SparkConnect*` subclasses, so narrowing further risks not retrying the very error we're guarding against. Also moved the `eventually` import up to the module-level imports.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57615][CONNECT][TESTS] Wait for the Connect server before creating a ResourceProfile in test_profile_before_sc_for_connect [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #56676:
URL: https://github.com/apache/spark/pull/56676#discussion_r3455813998
##########
python/pyspark/resource/tests/test_connect_resources.py:
##########
@@ -46,6 +46,17 @@ def test_profile_before_sc_for_connect(self):
).getOrCreate()
# Still can access taskResources, similar to executorResources.
self.assertEqual(rp.taskResources["cpus"].amount, 2.0)
+ # SPARK-57615: the local-cluster Spark Connect server can return errors on the first
+ # commands right after start-connect-server.sh returns, before its SparkContext is fully
+ # initialized. Wait until a trivial job succeeds before creating a ResourceProfile, which
+ # would otherwise race that initialization and fail server-side.
+ from pyspark.testing.utils import eventually
+
+ def _server_ready() -> bool:
+ spark.range(1).count()
+ return True
+
+ eventually(timeout=120, expected_exceptions=(Exception,))(_server_ready)()
Review Comment:
It runs a separate server so the output better has to be thrown to the console which should be easier. Anyway, this fix is both for debugging and actual fixing :-). Should be fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57615][CONNECT][TESTS] Wait for the Connect server before creating a ResourceProfile in test_profile_before_sc_for_connect [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on code in PR #56676:
URL: https://github.com/apache/spark/pull/56676#discussion_r3455788170
##########
python/pyspark/resource/tests/test_connect_resources.py:
##########
@@ -46,6 +46,17 @@ def test_profile_before_sc_for_connect(self):
).getOrCreate()
# Still can access taskResources, similar to executorResources.
self.assertEqual(rp.taskResources["cpus"].amount, 2.0)
+ # SPARK-57615: the local-cluster Spark Connect server can return errors on the first
+ # commands right after start-connect-server.sh returns, before its SparkContext is fully
+ # initialized. Wait until a trivial job succeeds before creating a ResourceProfile, which
+ # would otherwise race that initialization and fail server-side.
+ from pyspark.testing.utils import eventually
+
+ def _server_ready() -> bool:
+ spark.range(1).count()
+ return True
+
+ eventually(timeout=120, expected_exceptions=(Exception,))(_server_ready)()
Review Comment:
Oh so the dumped stack trace is not enough? `pystack` worked right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57615][CONNECT][TESTS] Wait for the Connect server before creating a ResourceProfile in test_profile_before_sc_for_connect [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #56676: [SPARK-57615][CONNECT][TESTS] Wait for the Connect server before creating a ResourceProfile in test_profile_before_sc_for_connect
URL: https://github.com/apache/spark/pull/56676
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57615][CONNECT][TESTS] Wait for the Connect server before creating a ResourceProfile in test_profile_before_sc_for_connect [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #56676:
URL: https://github.com/apache/spark/pull/56676#discussion_r3455744311
##########
python/pyspark/resource/tests/test_connect_resources.py:
##########
@@ -46,6 +46,17 @@ def test_profile_before_sc_for_connect(self):
).getOrCreate()
# Still can access taskResources, similar to executorResources.
self.assertEqual(rp.taskResources["cpus"].amount, 2.0)
+ # SPARK-57615: the local-cluster Spark Connect server can return errors on the first
+ # commands right after start-connect-server.sh returns, before its SparkContext is fully
+ # initialized. Wait until a trivial job succeeds before creating a ResourceProfile, which
+ # would otherwise race that initialization and fail server-side.
+ from pyspark.testing.utils import eventually
+
+ def _server_ready() -> bool:
+ spark.range(1).count()
+ return True
+
+ eventually(timeout=120, expected_exceptions=(Exception,))(_server_ready)()
Review Comment:
BTW, this is more fore debugging yet to identify why it is hanging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57615][CONNECT][TESTS] Wait for the Connect server before creating a ResourceProfile in test_profile_before_sc_for_connect [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #56676:
URL: https://github.com/apache/spark/pull/56676#issuecomment-4774795262
Merged to master and branch-4.x.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57615][CONNECT][TESTS] Wait for the Connect server before creating a ResourceProfile in test_profile_before_sc_for_connect [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on code in PR #56676:
URL: https://github.com/apache/spark/pull/56676#discussion_r3455132740
##########
python/pyspark/resource/tests/test_connect_resources.py:
##########
@@ -46,6 +46,17 @@ def test_profile_before_sc_for_connect(self):
).getOrCreate()
# Still can access taskResources, similar to executorResources.
self.assertEqual(rp.taskResources["cpus"].amount, 2.0)
+ # SPARK-57615: the local-cluster Spark Connect server can return errors on the first
+ # commands right after start-connect-server.sh returns, before its SparkContext is fully
+ # initialized. Wait until a trivial job succeeds before creating a ResourceProfile, which
+ # would otherwise race that initialization and fail server-side.
+ from pyspark.testing.utils import eventually
+
+ def _server_ready() -> bool:
+ spark.range(1).count()
+ return True
+
+ eventually(timeout=120, expected_exceptions=(Exception,))(_server_ready)()
Review Comment:
nit: if we have a more explicit exception to catch, it might be better than having `Exception` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org