[I] Two retry loops still hand-rolled: URLFetchUtil retries without delay, PythonProxyClient waits a constant delay [texera]

["aglinxinyuan (via GitHub)" <gi...@apache.org>]

aglinxinyuan opened a new issue, #7124:
URL: https://github.com/apache/texera/issues/7124

   ### Task Summary
   
   Two retry loops were left hand-rolled when #7119 consolidated the rest onto `RetryUtil.withBackoff`, because converting them changes runtime behavior rather than just structure. Each needs a decision before it can move.
   
   **1. `URLFetchUtil.getInputStreamFromURL`** (`common/workflow-operator`)
   
   ```scala
   def getInputStreamFromURL(urlObj: URL, retries: Int = 5): Option[InputStream]
   ```
   
   - Retries 5x with **no delay at all**, and returns `Option` rather than throwing.
   - Adopting the util's backoff means a dead URL costs ~3 s (200+400+800+1600) before giving up, where today it fails immediately. That is arguably the fix — an unthrottled 5x retry against a remote host is a small retry storm — but it changes how long a URL-scan operator takes to fail.
   - The `Option` return is a one-line adapter (`Try(...).toOption`), so the delay is the only real question.
   
   **2. `PythonProxyClient`'s Flight-connect loop** (`amber`)
   
   ```scala
   while (!connected && tryCount <= MAX_TRY_COUNT) { ... Thread.sleep(UNIT_WAIT_TIME_MS) ... }
   ```
   
   - Waits a **constant** `UNIT_WAIT_TIME_MS` between attempts, not a doubling one, so the util needs a delay-multiplier knob (`1` = constant) before it can host this.
   - Closes the Flight client between attempts; that per-attempt cleanup has to move into the operation body.
   - Throws `WorkflowRuntimeException` on give-up, where the util wraps in `RuntimeException`. Either the util grows a give-up hook, or this call site accepts the different type — worth deciding deliberately, since worker-bring-up failures are surfaced to users.
   
   Both are candidates, not obligations: if the answer for either is "the current behavior is what we want", closing this with that note recorded is a fine outcome.
   
   ### Task Type
   
   - [x] Refactor / Cleanup
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Two retry loops still hand-rolled: URLFetchUtil retries without delay, PythonProxyClient waits a constant delay [texera]

["anthonychengit (via GitHub)" <gi...@apache.org>]

anthonychengit commented on issue #7124:
URL: https://github.com/apache/texera/issues/7124#issuecomment-5655211656

   /take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org