You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dtaniwaki (via GitHub)" <gi...@apache.org> on 2026/06/14 08:20:41 UTC

[PR] [SPARK-57425][CONNECT][PYTHON] Make reattach iterator recover from short-TTL credential expiry [spark]

dtaniwaki opened a new pull request, #56497:
URL: https://github.com/apache/spark/pull/56497

   JIRA: https://issues.apache.org/jira/browse/SPARK-57425
   
   ### What changes were proposed in this pull request?
   
   This PR makes `ExecutePlanResponseReattachableIterator`
   (`python/pyspark/sql/connect/client/reattach.py`) survive short-TTL credential
   expiry. Two changes:
   
   **1. Per-RPC metadata refresh.** `metadata` now accepts a zero-arg `Callable`
   in addition to an `Iterable`. When a callable is supplied, the iterator
   invokes it immediately before every gRPC call:
   
   - initial `ExecutePlan`,
   - retry `ExecutePlan` after `INVALID_HANDLE.OPERATION_NOT_FOUND`,
   - `ReattachExecute`,
   - `ReleaseExecute` (`release_until` and `release_all`).
   
   `SparkConnectClient` now passes the bound method `self._builder.metadata`
   rather than calling it once at construction, so a custom `ChannelBuilder`
   that refreshes its parameters takes effect across reattach. Static iterables
   are materialized to a list so single-shot iterators stay usable. Provider
   failures inside `_release_*` log a distinct message to avoid being mistaken
   for server-side release errors.
   
   **2. One `PERMISSION_DENIED` retry per iterator.** A first mid-stream
   `PERMISSION_DENIED` is treated as a candidate token-expiry signal: the
   iterator clears its underlying stream and raises `RetryException`, mirroring
   the existing `DEADLINE_EXCEEDED` branch. The retry attempt issues
   `ReattachExecute`, the metadata provider is re-evaluated, and a refreshed
   credential takes effect. A second `PERMISSION_DENIED` per iterator
   propagates as a genuine auth failure.
   
   ### Why are the changes needed?
   
   The reattach mechanism exists to recover from a broken gRPC stream before
   `ResultComplete`. That recovery is structurally impossible today when the
   server enforces a short auth-token TTL (e.g. AWS Athena Spark, 30 min) and
   a query outlives the TTL:
   
   - the default retry policy does not treat `PERMISSION_DENIED` as retryable,
     so the 403 propagates before reattach is attempted, and
   - even if reattach did fire, it would carry the expired token captured at
     `__init__`.
   
   Both gaps must change for the iterator's contract to hold for this class
   of scenarios.
   
   This has stayed invisible in typical deployments because four conditions
   must align: a short server TTL, a stream that outlives it, a server that
   actively kills the stream on expiry, and reattach actually firing (no
   `ResultComplete`). Local dev without auth, on-prem with long-lived tokens,
   and short ad-hoc queries each violate at least one; managed
   federated-credential environments hit all four. `ChannelBuilder.metadata()`
   is already rebuilt per call, so non-reattach RPCs (`AnalyzePlan`, `Config`,
   ...) already pick up refreshed credentials — only the reattach iterator
   was stuck.
   
   The dbt-athena Spark adapter ships three runtime monkey-patches (stash the
   builder on the stub, refresh metadata before `ReattachExecute`, allow one
   `PERMISSION_DENIED` retry per iterator) that have been running in
   production. This PR folds those moving parts into upstream so the
   workaround becomes unnecessary.
   
   **Backport requested to branch-4.0, branch-4.1, branch-4.2.** Athena's
   Spark Connect endpoint and most current managed deployments run on the 4.x
   line, and 5.x adoption is realistically far off. The touched code is
   structurally identical across 4.x and master, so the backport is
   near-mechanical.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No. `ExecutePlanResponseReattachableIterator` is internal. Existing callers
   passing an iterable see no behavior change, and a `PERMISSION_DENIED` that
   does not stem from a recoverable mid-stream condition still propagates as
   before (the new retry budget is one per iterator).
   
   ### How was this patch tested?
   
   Two test classes in
   `python/pyspark/sql/tests/connect/client/test_client.py`, 10 new cases:
   
   - **`SparkConnectClientReattachTestCase`** — seven mock-stub cases using an
     extended `MockSparkConnectStub` that records the metadata of every RPC.
     Cover: per-RPC callable invocation, refresh on
     `INVALID_HANDLE.OPERATION_NOT_FOUND`, static-iterable caching, one-shot
     iterator materialization across a forced reattach, isolation of provider
     errors in `_release_*`, mid-stream `PERMISSION_DENIED` retry, and
     propagation after a second `PERMISSION_DENIED`.
   
   - **`SparkConnectClientReattachGrpcEndToEndTestCase`** — three cases that
     drive the iterator through a real in-process gRPC server (`grpc.server()`
     + a minimal `SparkConnectServiceServicer`) so the metadata callable,
     mid-stream `PERMISSION_DENIED` recovery, and the once-per-iterator retry
     budget are exercised at the wire level without a JVM or any external
     Spark process.
   
   Existing `SparkConnectClientReattachTestCase` and
   `SparkConnectClientRetriesTestCase` cases continue to pass.
   
   ```
   python -m pytest python/pyspark/sql/tests/connect/client/test_client.py \
                    python/pyspark/sql/tests/connect/client/test_client_retries.py
   ```
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code (Anthropic), Opus 4.7 (1M context)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57425][CONNECT][PYTHON] Make reattach iterator recover from short-TTL credential expiry [spark]

Posted by "shrirangmhalgi (via GitHub)" <gi...@apache.org>.
shrirangmhalgi commented on code in PR #56497:
URL: https://github.com/apache/spark/pull/56497#discussion_r3415193959


##########
python/pyspark/sql/connect/client/reattach.py:
##########
@@ -295,7 +323,7 @@ def _call_iter(self, iter_fun: Callable) -> Any:
                 self._iterator = iter(
                     self._stub.ExecutePlan(
                         self._initial_request,
-                        metadata=self._metadata,
+                        metadata=self._metadata_provider(),

Review Comment:
   Nit (Non Blocking): `_permission_denied_retried` isn't reset when `INVALID_HANDLE.OPERATION_NOT_FOUND` triggers a fresh ExecutePlan. If the sequence is `PERMISSION_DENIED → OPERATION_NOT_FOUND → fresh ExecutePlan → PERMISSION_DENIED`, the second denial propagates even though the stream is logically new. Probably fine as-is (conservative budget design), just noting for completeness.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57425][CONNECT][PYTHON] Make reattach iterator recover from short-TTL credential expiry [spark]

Posted by "dtaniwaki (via GitHub)" <gi...@apache.org>.
dtaniwaki commented on PR #56497:
URL: https://github.com/apache/spark/pull/56497#issuecomment-4710474167

   @shrirangmhalgi Thank you for reviewing the PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57425][CONNECT][PYTHON] Make reattach iterator recover from short-TTL credential expiry [spark]

Posted by "shrirangmhalgi (via GitHub)" <gi...@apache.org>.
shrirangmhalgi commented on PR #56497:
URL: https://github.com/apache/spark/pull/56497#issuecomment-4710352209

   LGTM. I had a parallel PR (#56496) for the same issue - closed #56496 in favor of this one. 
   
   The retry budget approach is cleaner than global retryable status in `retries.py`, and the callable metadata pattern matches what I had independently. Nice work.
   
   cc @hvanhovell


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57425][CONNECT][PYTHON] Make reattach iterator recover from short-TTL credential expiry [spark]

Posted by "dtaniwaki (via GitHub)" <gi...@apache.org>.
dtaniwaki commented on code in PR #56497:
URL: https://github.com/apache/spark/pull/56497#discussion_r3627227308


##########
python/pyspark/sql/connect/client/reattach.py:
##########
@@ -310,6 +339,19 @@ def _call_iter(self, iter_fun: Callable) -> Any:
                 )
                 self._iterator = None
                 raise RetryException() from e
+            elif (
+                e.code() == grpc.StatusCode.PERMISSION_DENIED
+                and not self._permission_denied_retried
+            ):
+                # Treat the first mid-stream PERMISSION_DENIED as a token-expiry signal; reattach
+                # once so the metadata provider can produce a fresh credential.
+                logger.debug(
+                    f"PERMISSION_DENIED on stream for operation {self._operation_id}; "
+                    f"allowing one reattach with refreshed metadata."
+                )
+                self._permission_denied_retried = True
+                self._iterator = None
+                raise RetryException() from e

Review Comment:
   @uros-b Could you tell me what else I can prepare to get this PR merged or reviewed?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57425][CONNECT][PYTHON] Make reattach iterator recover from short-TTL credential expiry [spark]

Posted by "dtaniwaki (via GitHub)" <gi...@apache.org>.
dtaniwaki commented on code in PR #56497:
URL: https://github.com/apache/spark/pull/56497#discussion_r3450038714


##########
python/pyspark/sql/connect/client/reattach.py:
##########
@@ -310,6 +339,19 @@ def _call_iter(self, iter_fun: Callable) -> Any:
                 )
                 self._iterator = None
                 raise RetryException() from e
+            elif (
+                e.code() == grpc.StatusCode.PERMISSION_DENIED
+                and not self._permission_denied_retried
+            ):
+                # Treat the first mid-stream PERMISSION_DENIED as a token-expiry signal; reattach
+                # once so the metadata provider can produce a fresh credential.
+                logger.debug(
+                    f"PERMISSION_DENIED on stream for operation {self._operation_id}; "
+                    f"allowing one reattach with refreshed metadata."
+                )
+                self._permission_denied_retried = True
+                self._iterator = None
+                raise RetryException() from e

Review Comment:
   Maybe, I should revert the commit first to make it at least safer?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57425][CONNECT][PYTHON] Make reattach iterator recover from short-TTL credential expiry [spark]

Posted by "vgkowski (via GitHub)" <gi...@apache.org>.
vgkowski commented on PR #56497:
URL: https://github.com/apache/spark/pull/56497#issuecomment-5172030458

   any progress here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57425][CONNECT][PYTHON] Make reattach iterator recover from short-TTL credential expiry [spark]

Posted by "shrirangmhalgi (via GitHub)" <gi...@apache.org>.
shrirangmhalgi commented on PR #56497:
URL: https://github.com/apache/spark/pull/56497#issuecomment-4714432042

   > @shrirangmhalgi Thank you for reviewing the PR and sorry for the duplicate work!
   
   No worries at all - happy it converged on the better design. Hope it gets merged soon!
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57425][CONNECT][PYTHON] Make reattach iterator recover from short-TTL credential expiry [spark]

Posted by "dtaniwaki (via GitHub)" <gi...@apache.org>.
dtaniwaki commented on code in PR #56497:
URL: https://github.com/apache/spark/pull/56497#discussion_r3415293039


##########
python/pyspark/sql/connect/client/reattach.py:
##########
@@ -295,7 +323,7 @@ def _call_iter(self, iter_fun: Callable) -> Any:
                 self._iterator = iter(
                     self._stub.ExecutePlan(
                         self._initial_request,
-                        metadata=self._metadata,
+                        metadata=self._metadata_provider(),

Review Comment:
   Fixed https://github.com/apache/spark/pull/56497/commits/2fe61b652281544639c4ea546bf94cd63b815070



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57425][CONNECT][PYTHON] Make reattach iterator recover from short-TTL credential expiry [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #56497:
URL: https://github.com/apache/spark/pull/56497#discussion_r3430520073


##########
python/pyspark/sql/connect/client/reattach.py:
##########
@@ -310,6 +339,19 @@ def _call_iter(self, iter_fun: Callable) -> Any:
                 )
                 self._iterator = None
                 raise RetryException() from e
+            elif (
+                e.code() == grpc.StatusCode.PERMISSION_DENIED
+                and not self._permission_denied_retried
+            ):
+                # Treat the first mid-stream PERMISSION_DENIED as a token-expiry signal; reattach
+                # once so the metadata provider can produce a fresh credential.
+                logger.debug(
+                    f"PERMISSION_DENIED on stream for operation {self._operation_id}; "
+                    f"allowing one reattach with refreshed metadata."
+                )
+                self._permission_denied_retried = True
+                self._iterator = None
+                raise RetryException() from e

Review Comment:
   Posting a drive-by note here: PERMISSION_DENIED raises RetryException, which retries with no backoff and never consumes retry budget (retries.py:233,258). Combined with the latch reset on the OPERATION_NOT_FOUND path (line 330), a token that can't be refreshed to a valid one spins forever with no forward progress: ExecutePlan → PERMISSION_DENIED → ReattachExecute → OPERATION_NOT_FOUND → fresh ExecutePlan (resets latch) → PERMISSION_DENIED → …, hammering token-refresh + 2 RPCs per lap.
   
   The mock tests only terminate because the scripted ops run out. This is also a regression - today this case fails fast. Suggest a hard per-iterator cap (counter, not a boolean reset on every response/OPERATION_NOT_FOUND) and/or real backoff. cc @HyukjinKwon @zhengruifeng who have more context to investigate this further.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57425][CONNECT][PYTHON] Make reattach iterator recover from short-TTL credential expiry [spark]

Posted by "dtaniwaki (via GitHub)" <gi...@apache.org>.
dtaniwaki commented on code in PR #56497:
URL: https://github.com/apache/spark/pull/56497#discussion_r3456807870


##########
python/pyspark/sql/connect/client/reattach.py:
##########
@@ -310,6 +339,19 @@ def _call_iter(self, iter_fun: Callable) -> Any:
                 )
                 self._iterator = None
                 raise RetryException() from e
+            elif (
+                e.code() == grpc.StatusCode.PERMISSION_DENIED
+                and not self._permission_denied_retried
+            ):
+                # Treat the first mid-stream PERMISSION_DENIED as a token-expiry signal; reattach
+                # once so the metadata provider can produce a fresh credential.
+                logger.debug(
+                    f"PERMISSION_DENIED on stream for operation {self._operation_id}; "
+                    f"allowing one reattach with refreshed metadata."
+                )
+                self._permission_denied_retried = True
+                self._iterator = None
+                raise RetryException() from e

Review Comment:
   @shrirangmhalgi @uros-b 
   I fixed the code to handle the infinite loop case. https://github.com/apache/spark/pull/56497/commits/ae19ecee0cd1261bdffd01a798920a06ccabf76d
   
   Could you review it again?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org