You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2026/08/26 20:52:09 UTC

[PR] [SPARK-58987][INFRA][FOLLOWUP] Stop the arrow-python-udf parity file hanging on CI [spark-connect-rust]

HyukjinKwon opened a new pull request, #58:
URL: https://github.com/apache/spark-connect-rust/pull/58

   ### What changes were proposed in this pull request?
   
   Start the CI Spark Connect server with
   `spark.connect.execute.reattachable.senderMaxStreamDuration=1200s` in the
   Python-Connect-parity workflow.
   
   ### Why are the changes needed?
   
   After #54 merged, `test_parity_arrow_python_udf.py` failed the master parity gate with
   a whole-file **TIMEOUT (p=0)** — zero tests completed in the 1200s per-file cap — even
   though the exact same file passes **deterministically locally in ~113s (320 passed, 64
   skipped)**, matching the reference client's ~120s. So it is not a client correctness bug
   and not mere slowness (it ran the full 1200s doing nothing).
   
   Root cause: on the loaded single-node CI server, a slow first arrow-Python-UDF worker
   lets the default `senderMaxStreamDuration` elapse, so the server ends the reattachable
   `ExecutePlan` response stream early and the client must `ReattachExecute`; that reattach
   path stalls under these conditions and hangs the whole file. Raising the sender duration
   keeps the stream open until results are produced, so no reattach is triggered and the
   file completes normally. (The underlying reattach-path robustness is worth hardening in
   the client separately; this unblocks the gate without masking a client bug.)
   
   ### Does this PR introduce _any_ user-facing change?
   
   No — CI configuration only.
   
   ### How was this patch tested?
   
   CI: the Python Connect parity job on this PR. Locally the file already passes in ~113s.
   
   This pull request and its description were written by Isaac.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59032][CORE] Apply gRPC keepalive so a wedged Connect stream recovers instead of hanging [spark-connect-rust]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #58:
URL: https://github.com/apache/spark-connect-rust/pull/58#issuecomment-5434046573

   Thanks @viirya — thorough review, all fair. Addressed in the latest commit (866acf0):
   
   - **keepalive timeout coercion.** Fixed the real bug: `.max(0)` is gone. The logic now lives in a validated `ChannelBuilder::keepalive()` that returns `None` (never a zero-duration) when the interval *or* timeout is non-positive — so `grpc_keepalive_timeout_ms=-1` disables keepalive, consistent with how a `-1` interval already behaves, instead of producing an immediately-expiring PING. Added a unit test for exactly the (valid-interval, `-1`-timeout) combination.
   - **max message length (128 MiB parsed but never applied).** Agreed it's the same bug-class and a real >4 MiB decode hazard. Filed as a follow-up: **SPARK-59037**, so it isn't rediscovered as a mystery decode error.
   - **diagnostic layer in the description.** Folded the faulthandler watchdog + `_dump_server_jvm` into the PR description explicitly. Also tightened all three: `found[-1]` instead of `[0]`, raised the jstack cap 20k→60k (the first run truncated exactly the execute/PythonRunner threads), a comment that the JVM dump is only unambiguous at `--jobs 1`, and a try/except around the dump-file open so an unwritable `RUST_HANG_DUMP` can't fail plugin import.
   - **two root causes / framing.** Fully agree. Reframed both the PR and SPARK-59032: keepalive makes the symptom *recoverable* but does **not** explain "server produced nothing while the reference got an error in <1s," so the ticket stays open for the true cause (the added jstack targets exactly that). Noted the `FLAKY_FILES` retry is a 360s-cap-per-attempt trade — coverage over speed.
   - **smaller things.** Removed the duplicated lazy-connect comment, reverted the `skipped_file` `"env"` string back to a bool, switched to inline `{e}` captures, and trimmed the keepalive comment to the *why*.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59032][CORE] Fix arrow-parity hang: bound ReleaseExecute so HTTP/2 streams can't accumulate [spark-connect-rust]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #58:
URL: https://github.com/apache/spark-connect-rust/pull/58#issuecomment-5446584126

   Update: the actual root cause is now found and fixed, and the PR has pivoted accordingly (title/description updated).
   
   It was **not** the keepalive path and **not** test-specific. Via CI RPC-tracing + a server jstack: the reattachable-execute driver fires a `ReleaseExecute` after every response on a background pool, and each release closure captures the operation's iterator. Our `release_execute` was **unbounded**, so a stalled release kept its future pending → kept the iterator alive → kept the iterator's open `ExecutePlan` HTTP/2 stream alive; enough of those accumulate on the single shared connection that a later `ExecutePlan` can't get a stream and the client blocks forever while the server is fully idle. Cumulative → it landed on different tests depending on timing (`test_raise_stop_iteration`, `test_udf_in_generate`, …).
   
   Fix: bound `release_execute` with a 30s deadline. `ReleaseExecute` is best-effort — reattach.py runs it on a pool and logs-and-ignores failures ("server is equipped to deal with abandoned executions", identical on v4.2.0 and master) — so a timed-out release frees its thread and closes its stream, nothing accumulates. Verified in CI: with the hang test **not** deselected, `test_parity_arrow_python_udf.py` passes 319/320 and all arrow files pass. The keepalive change (which you reviewed) stays as an independent robustness fix; all the earlier workarounds are removed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59032][INFRA] Run behavioral e2e in a separate job so the parity gate gets a clean Connect server [spark-connect-rust]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #58: [SPARK-59032][INFRA] Run behavioral e2e in a separate job so the parity gate gets a clean Connect server
URL: https://github.com/apache/spark-connect-rust/pull/58


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org