Re: [I] Airflow 3.2 Worker Failures on TaskInstance Finish (HTTP 409 invalid_state) [airflow]

["KeeganCarter11 (via GitHub)" <gi...@apache.org>]

KeeganCarter11 commented on issue #65708:
URL: https://github.com/apache/airflow/issues/65708#issuecomment-4406353403

   I patched this issue my MWAA Local runner in a kinda ugly way - I'm sure that someone with more experience can take this further.
   
   inside `airflow/sdk/api/client.py` I replaced the floowing
   ```python
    except ServerResponseError as e:
               if e.response.status_code == HTTPStatus.CONFLICT:
                   detail = e.detail
                   if (
                       isinstance(detail, dict)
                       and detail.get("reason") == "invalid_state"
                       and detail.get("previous_state") == "running"
                   ):
                       raise TaskAlreadyRunningError(f"Task instance {id} is already running") from e
               raise
   ```
   
   with 
   ```python
   except ServerResponseError as e:
               if e.response.status_code in (HTTPStatus.CONFLICT, HTTPStatus.NOT_FOUND):
                   # LOCAL PATCH: treat any 409 or 404 as TaskAlreadyRunningError.
                   # SQS at-least-once delivery can cause duplicate messages. The original
                   # code only handles previous_state=running 409s, but duplicate workers
                   # can also see 409 with previous_state=success/failed, or 404 if the
                   # task instance was already cleaned up. All of these should be ignored.
                   raise TaskAlreadyRunningError(f"Task instance {id} is not in a startable state") from e
               raise
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org