You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "amoghrajesh (via GitHub)" <gi...@apache.org> on 2026/08/06 04:47:11 UTC

[PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

amoghrajesh opened a new pull request, #71211:
URL: https://github.com/apache/airflow/pull/71211

    <!-- SPDX-License-Identifier: Apache-2.0
         https://www.apache.org/licenses/LICENSE-2.0 -->
   
   <!--
   Thank you for contributing!
   
   Please provide above a brief description of the changes made in this pull request.
   Write a good git commit message following this guide: https://chris.beams.io/posts/git-commit/
   
   Please make sure that your code changes are covered with tests.
   And in case of new features or big changes remember to adjust the documentation.
   
   For user-facing UI changes, please attach before/after screenshots (or a short
   screen recording) so reviewers can assess the visual impact.
   
   Feel free to ping (in general) for the review if you do not see reaction for a few days
   (72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
   
   In case of an existing issue, reference it using one of the following:
   
   * closes: #ISSUE
   * related: #ISSUE
   -->
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   <!--
   If generative AI tooling has been used in the process of authoring this PR, please
   change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
   -->
   
   - [x] Yes - claude sonnet
   
   <!--
   Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   -->
   
   ## Summary
   
   This PR intends to extend GlueJobOperator for durable execution (AIP-103): on Airflow 3.3+, the job
   run id is persisted to task state store immediately after submission, so a worker crash and retry reconnects to the run already executing in Glue instead of starting a new one.
   
   ## How it works today
   
   GlueJobOperator's resume_glue_job_on_retry mechanism has two tiers: an XCom cached run id (fast path), falling back to a paginated scan of the job's run history tagged by task UUID. I reproduced twice against real AWS Glue that the fast path never fires on Airflow 3.x: the server clears a task instance's XComs before every non deferral attempt, so the cached id is always gone by the time the next retry looks for it. Every retry was paying for the full scan regardless, and the scan itself returned the first UUID match regardless of state, so an earlier terminated run could mask a active one underneath it.
   
   ## Why
   
   `resume_glue_job_on_retry` already tries to do this via XCom, falling back to
   a scan of the job's run history when XCom is empty. Confirmed against real
   AWS Glue (killed the worker mid-poll, twice, under two different kill
   methods) that the XCom fast path never fires on Airflow 3.x: the server
   clears a task instance's XComs before every non-deferral attempt
   (`task_instances.py`'s `xcom_keys_to_clear`), so by the time the next retry
   looks for the cached id it's already gone. Every retry pays for the full
   paginated scan regardless of timing, not just the narrow crash-before-push
   window the mechanism's original PR (#62560) was written to cover.
   
   The scan itself also has a latent bug: it returns the first task-UUID match
   regardless of state, so an earlier terminated run can mask a still-active
   one further down the page. Not fixed in this PR -- see Scope below.
   
   This matters more for Glue than for most `ResumableJobMixin` ports: a Glue
   job's `concurrent_run_limit` defaults to `1`, so a duplicate submission
   doesn't just waste compute, it fails outright with
   `ConcurrentRunsExceededException` and the task keeps retrying against a run
   it can never observe.
   
   ## What changed
   
   - `GlueJobOperator` now inherits `ResumableJobMixin`. On Airflow 3.3+, the mixin checks task state store before doing anything else: an active run reconnects directly (one `get_job_state` call), a succeeded or externally stopped run returns without resubmitting, a terminal run resubmits fresh.
   - The **existing** XCom followed by scan mechanism is kept exactly as it was, relocated into `submit_job` (the mixin only calls `submit_job` when task state has nothing usable). This is what Airflow <3.3 runs on entirely, and what 3.3+ falls back to as a one-time bootstrap for a run started before this upgrade, or before task state store had anything persisted.
   - `resume_glue_job_on_retry` is deprecated in favour of `durable` (`AirflowProviderDeprecationWarning`, gated to fire only on Airflow 3.3+); passing it still works and maps onto `durable`.
   - A run `STOPPED` outside Airflow (e.g. cancelled manually) is treated as a success rather than resubmitted -- the work is genuinely finished, just not the way the task expected -- with a warning logged. Matches how `GlueJobHook.job_completion` and `GlueJobCompleteTrigger` already treat `STOPPED`.
   - Deferrable takes precedence over durable, unchanged in spirit from today: the Triggerer already tracks the run across the wait, so the run id isn't persisted to task state on that path.
   
   ## Behaviour change
   
   `durable` is owned by the mixin and defaults to `True`. `resume_glue_job_on_retry` defaulted to `False`. A Dag that never touched the flag goes from "always resubmit on retry" to "reattach on retry" with no code change.
   
   Documented in `providers/amazon/docs/changelog.rst`
   
   ## Scope
   
   Deliberately left alone, all pre existing, none introduced by this PR:
   
   - The scan's first match regardless of state bug described above in the PR.
   - `GlueJobHook._handle_state` treats only `FAILED`/`TIMEOUT` as failure, so AWS's terminal `ERROR`/`EXPIRED` states fall through to "keep polling" and the sync poll loop never exits on them.
   - `ResumableJobMixin.execute_resumable`'s reconnect path returns `poll_until_complete`'s `None` instead of calling `get_job_result`, unlike the other two paths. Not touched here (no task-sdk change) -- every method instead sets `self._job_run_id`, and `execute()` returns that directly.
   
   ## Why `task_state_store` is a better crash recovery mechanism
   
   **1. It survives retries while xcom does not.**
   Airflow clears all xcoms before each retry attempt. `task_state_store` is built specifically to persist across retries.
   
   **2. One call vs. a full scan.**
   Store hit -> single `get_job_state` call. Scan -> paginates the entire job run history, 50 at a time.
   
   **3. No ambiguity.**
   Store returns the exact id you saved. The scan matches by tag and can pick a stale run if an older one shares the tag.
   
   
   
   ## Testing
   
   Uploaded this script on AWS Glue and created an _ETL JOB_ there:
   ```python
   import sys
   import time
   
   
   def get_arg(name, default=None):
       flag = f"--{name}"
       if flag in sys.argv:
           i = sys.argv.index(flag)
           if i + 1 < len(sys.argv):
               return sys.argv[i + 1]
       return default
   
   
   task_uuid = get_arg("airflow_task_uuid", "<NOT INJECTED>")
   sleep_total = int(get_arg("sleep_seconds", "600"))
   
   print(f"argv: {sys.argv}", flush=True)
   print(f"airflow_task_uuid = {task_uuid}", flush=True)
   print(f"sleeping for {sleep_total}s", flush=True)
   
   elapsed = 0
   while elapsed < sleep_total:
       time.sleep(15)
       elapsed += 15
       print(f"alive {elapsed}/{sleep_total}s uuid={task_uuid}", flush=True)
   
   print("done", flush=True)
   
   ```
   
   <img width="2559" height="1148" alt="image" src="https://github.com/user-attachments/assets/e55604b8-064d-4d7b-832a-718f42e7dba5" />
   
   
   Using this DAG:
   ```python
   from datetime import datetime, timedelta
   
   from airflow.providers.amazon.aws.operators.glue import GlueJobOperator
   from airflow.sdk import DAG
   
   with DAG(
       dag_id="glue_job_trial",
       schedule=None,
       start_date=datetime(2024, 1, 1),
       catchup=False,
   ) as dag:
       run_glue = GlueJobOperator(
           task_id="run_glue",
           job_name="airflow_durable_test",
           aws_conn_id="aws_default",
           region_name="us-east-1",
           durable=True,
           script_args={"--sleep_seconds": "900"},
           wait_for_completion=True,
           deferrable=False,
           verbose=True,
           retries=3,
           retry_delay=timedelta(seconds=20),
       )
   
   ```
   
   ### Case 1: Crash Recovery after worker crash
   
   Launched the DAG:
   
   <img width="2559" height="1148" alt="image" src="https://github.com/user-attachments/assets/2e377d09-ba24-4410-9dbd-138b9259c4cf" />
   
   
   
   Glue Job:
   
   <img width="2559" height="1148" alt="image" src="https://github.com/user-attachments/assets/e707d43a-0112-4234-b24e-b27f9ff45e34" />
   
   
   State Store:
   
   <img width="2559" height="1148" alt="image" src="https://github.com/user-attachments/assets/1fe0f914-0cd8-44ac-8169-661825230af5" />
   
   
   Brought the worker down, see that the airflow task instance goes to retry state
   
   <img width="2559" height="1148" alt="image" src="https://github.com/user-attachments/assets/121885f1-6005-4496-ba8d-fc51c118f7fd" />
   
   
   Reconnects to same job
   
   <img width="2559" height="1148" alt="image" src="https://github.com/user-attachments/assets/f498ad7d-bae2-4469-941c-f06fb0d5800e" />
   
   
   ### Case 2: Stop a run which was submitted from Glue Console, there should be no resubmit
   
   <img width="2559" height="1148" alt="image" src="https://github.com/user-attachments/assets/af71dbcd-612b-4884-9e23-3f8c7cd5a959" />
   
   
   <img width="2559" height="1148" alt="image" src="https://github.com/user-attachments/assets/781c7ea9-5a75-433e-881d-9de2d4f38d2f" />
   
   
   <img width="2559" height="1191" alt="image" src="https://github.com/user-attachments/assets/d28b19bb-fd57-4d9f-92f1-ebc92fa538a0" />
   
   
   ### Case 3: `wait_for_completion=False` persists to state store
   
   Update task to:
   ```python
       run_glue = GlueJobOperator(
           task_id="run_glue",
           job_name="airflow_durable_test",
           aws_conn_id="aws_default",
           region_name="us-east-1",
           durable=True,
           script_args={"--sleep_seconds": "900"},
           wait_for_completion=False,
           deferrable=False,
           verbose=True,
           retries=3,
           retry_delay=timedelta(seconds=20),
       )
   ```
   
   <img width="2559" height="1191" alt="image" src="https://github.com/user-attachments/assets/906035d1-0f5e-47a1-83be-76453e5b90af" />
   
   
   <img width="2559" height="1191" alt="image" src="https://github.com/user-attachments/assets/b36f16da-0498-4cb7-bcfc-1b2a81fd66f5" />
   
   
   
   
   
   
   
   
   
   
   
   ---
   
   * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
   * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3756671370


##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -139,6 +139,79 @@ To submit a new AWS Glue job you can use :class:`~airflow.providers.amazon.aws.o
   The same AWS IAM role used for the crawler can be used here as well, but it will need
   policies to provide access to the output location for result data.
 
+Durable execution
+==================
+
+``GlueJobOperator`` submits a job run and then polls it to completion on the worker. By default
+the operator runs in a *durable* mode that makes this crash-safe: the Glue job run id is
+persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>` before
+polling begins, so if the worker crashes or is preempted and the task is retried, the operator
+reconnects to the run that is already executing in Glue instead of starting a new one.
+
+This matters more for Glue because a Glue job's ``concurrent_run_limit`` defaults to ``1``, so
+submitting a second run while the first is still active does not create a harmless duplicate, it
+fails outright with ``ConcurrentRunsExceededException`` and the task keeps retrying against a run
+it can never see. Durable execution turns that retry into a normal reconnect.
+
+On retry the operator checks the prior run's state:
+
+* if it is still starting, running, waiting for capacity, or being stopped, the operator
+  reconnects and continues polling
+* if it already succeeded, the operator returns immediately without resubmitting
+* if it stopped, failed terminally, or its id has expired and is no longer found, the operator
+  submits the job fresh
+
+A stopped run resubmits rather than being treated as a success. Glue's API has no way to tell a
+run that was cancelled manually (for example, in the AWS console) apart from one this operator's
+own :meth:`~airflow.providers.amazon.aws.operators.glue.GlueJobOperator.on_kill` stopped, which
+happens whenever ``stop_job_run_on_kill=True`` and the task is killed -- on SIGTERM, on
+``execution_timeout``, or when the task is cleared while running. Since the two cases can't be
+told apart, a stopped run always resubmits, so a self-inflicted stop never gets silently reported
+as a false success.
+
+This protection also applies when ``wait_for_completion=False`` -- even though that task attempt
+never polls at all, a retry after a successful submission still reconnects rather than
+resubmitting, since the run id is persisted immediately after submission regardless of whether the
+task waits for it to finish.
+
+Durable execution requires Airflow 3.3 or newer for the task state store lookup above. On earlier
+Airflow versions, or if the task state store is unavailable at runtime, ``durable=True`` still
+recovers a prior run, just via an older mechanism: the operator checks XCom for a cached run id
+first, then falls back to scanning the job's run history for a run tagged with this task
+instance's identity, and reconnects if it finds one that is still active.
+
+Like the persisted state itself, the stored run id isn't deleted automatically, that only happens
+when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is longer than
+``[state_store] default_retention_days`` (30 days by default) and cleanup runs in between, the run
+id won't be there for the next retry, and the operator falls back to the XCom/scan mechanism
+above rather than reconnecting via task state store. Avoid running cleanup on a schedule shorter
+than your longest ``retry_delay``.
+
+Clearing a task is treated the same as a retry, which matters specifically for a task whose job
+already succeeded: clearing does not delete the stored run id, so the next attempt reads it back
+and returns immediately without submitting anything to Glue. See
+:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
+``[state_store] clear_on_success`` setting that restores "clearing always resubmits."
+
+To opt out and always start a fresh run on retry, set ``durable=False``:
+
+.. code-block:: python
+
+  glue_job = GlueJobOperator(
+      task_id="glue_job",
+      job_name="my_glue_job",
+      script_location="s3://glue-examples/glue-scripts/sample_aws_glue_job.py",
+      durable=False,
+  )
+
+Durable execution applies to the synchronous path. When ``deferrable=True`` is set, the Triggerer
+already tracks the run across the wait, so deferrable mode takes precedence and ``durable`` has no

Review Comment:
   You were right, restored it. `durable` now tags every attempt (deferrable included) with the task's identity, and a deferrable retry scans for that tag and reattaches instead of resubmitting into `ConcurrentRunsExceededException`, matching what main did. Docs updated to describe this instead of "durable has no effect on deferrable."



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3765349123


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -158,6 +203,16 @@ def __init__(
         ),
         **kwargs,
     ):
+        if resume_glue_job_on_retry is not None:
+            # Kept as a real named parameter (not **kwargs) so `default_args` still applies correctly.
+            if AIRFLOW_V_3_3_PLUS:

Review Comment:
   Fixed, and KPO had the identical bug which I am fixing too.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3747218490


##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -139,6 +139,70 @@ To submit a new AWS Glue job you can use :class:`~airflow.providers.amazon.aws.o
   The same AWS IAM role used for the crawler can be used here as well, but it will need
   policies to provide access to the output location for result data.
 
+Durable execution
+==================
+
+``GlueJobOperator`` submits a job run and then polls it to completion on the worker. By default
+the operator runs in a *durable* mode that makes this crash-safe: the Glue job run id is
+persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>` before
+polling begins, so if the worker crashes or is preempted and the task is retried, the operator
+reconnects to the run that is already executing in Glue instead of starting a new one.
+
+This matters more for Glue because a Glue job's ``concurrent_run_limit`` defaults to ``1``, so
+submitting a second run while the first is still active does not create a harmless duplicate, it
+fails outright with ``ConcurrentRunsExceededException`` and the task keeps retrying against a run
+it can never see. Durable execution turns that retry into a normal reconnect.
+
+On retry the operator checks the prior run's state:
+
+* if it is still starting, running, waiting for capacity, or being stopped, the operator
+  reconnects and continues polling
+* if it already succeeded, or was stopped outside Airflow, the operator returns immediately
+  without resubmitting

Review Comment:
   Handled in 85398bfb29



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +363,146 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        if self.durable:
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])
+            if existing_job_run_id:
+                self._set_job_run_id(context, existing_job_run_id)
+                return existing_job_run_id
+        self.log.info(
+            "Initializing AWS Glue Job: %s. Wait for completion: %s",
+            self.job_name,
+            self.wait_for_completion,
+        )
+        glue_job_run = self.hook.initialize_job(script_args, self.run_job_kwargs)
+        # Set immediately (before any polling) so on_kill can stop the run even if the worker dies
+        # before poll_until_complete runs.
+        self._set_job_run_id(context, glue_job_run["JobRunId"])
+        return glue_job_run["JobRunId"]
+
+    def get_job_status(self, external_id: JsonValue, context: Context) -> str:
+        """Query the raw job run state; a run id Glue no longer knows about degrades to NOT_FOUND."""
+        job_run_id = cast("str", external_id)
+        # This is the first place a reconnecting attempt learns the run id.
+        self._set_job_run_id(context, job_run_id)
+        try:
+            return self.hook.get_job_state(self.job_name, job_run_id)
+        except ClientError as e:
+            if e.response["Error"]["Code"] == "EntityNotFoundException":
+                return "NOT_FOUND"
+            raise
+
+    def is_job_active(self, status: str) -> bool:
+        return status not in (*JOB_RUN_TERMINAL_STATES, NOT_FOUND_STATE)
+
+    def is_job_succeeded(self, status: str) -> bool:
+        if status == "STOPPED":

Review Comment:
   Handled in 85398bfb29



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +363,146 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        if self.durable:
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])
+            if existing_job_run_id:
+                self._set_job_run_id(context, existing_job_run_id)
+                return existing_job_run_id
+        self.log.info(
+            "Initializing AWS Glue Job: %s. Wait for completion: %s",
+            self.job_name,
+            self.wait_for_completion,
+        )
+        glue_job_run = self.hook.initialize_job(script_args, self.run_job_kwargs)
+        # Set immediately (before any polling) so on_kill can stop the run even if the worker dies
+        # before poll_until_complete runs.
+        self._set_job_run_id(context, glue_job_run["JobRunId"])

Review Comment:
   Handled in 85398bfb29



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +363,146 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        if self.durable:
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])

Review Comment:
   Handled in 85398bfb29



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3765509560


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +364,147 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """Record the run id and surface its console link; guarded so one attempt logs it once."""
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first, then falls back to a task-UUID scan. The XCom tier
+        only works on Airflow 2.11-3.2; Airflow 3.3+ clears task XComs before every non-deferral
+        attempt, so it always misses there and the scan runs every time.
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def _has_stored_external_id(self, context: Context) -> bool:
+        task_state_store = context.get("task_state_store")
+        return task_state_store is not None and task_state_store.get(self.external_id_key) is not None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        # Scan only when there's nothing else to go on: first attempt, no store, or a store that
+        # never recorded this key. A stored id (even terminal) means the caller already decided.
+        if self.durable and context["ti"].try_number > 1 and not self._has_stored_external_id(context):
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])

Review Comment:
   I figured that I can just make it behave like earlier for airflow versions less than 3.3, and thats what I did so that this overhead is not a surprise for sonmeone upgrading their providers. 
   
   - On <3.3, the scan is back to being purely opt-in, same as main before this PR ever existed.
   - On 3.3+, it behaves as durability is on by default
   
   Opened #71489 to track the underlying issues (unbounded pagination, silent swallow, broad except Exception, missing IAM doc)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3765751815


##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -139,6 +139,86 @@ To submit a new AWS Glue job you can use :class:`~airflow.providers.amazon.aws.o
   The same AWS IAM role used for the crawler can be used here as well, but it will need
   policies to provide access to the output location for result data.
 
+Durable execution
+==================
+
+``GlueJobOperator`` submits a job run and then polls it to completion on the worker. By default
+the operator runs in a *durable* mode that makes this crash-safe: the Glue job run id is
+persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>` before
+polling begins, so if the worker crashes or is preempted and the task is retried, the operator
+reconnects to the run that is already executing in Glue instead of starting a new one.
+
+This matters more for Glue because a Glue job's ``concurrent_run_limit`` defaults to ``1``, so
+submitting a second run while the first is still active does not create a harmless duplicate, it
+fails outright with ``ConcurrentRunsExceededException`` and the task keeps retrying against a run
+it can never see. Durable execution turns that retry into a normal reconnect.
+
+On retry the operator checks the prior run's state:
+
+* if it is still starting, running, waiting for capacity, or being stopped, the operator
+  reconnects and continues polling
+* if it already succeeded, the operator returns immediately without resubmitting
+* if it failed terminally, or its id has expired and is no longer found, the operator submits the
+  job fresh
+* if it already stopped, the operator submits the job fresh
+
+A stopped run is treated as a failure, not a success. Glue's API has no way to tell a run
+cancelled manually (for example, in the AWS console) apart from one this operator's own
+:meth:`~airflow.providers.amazon.aws.operators.glue.GlueJobOperator.on_kill` stopped, which happens
+whenever ``stop_job_run_on_kill=True`` and the task is killed -- on SIGTERM, on
+``execution_timeout``, or when the task is cleared while running. If the stored state is already
+``STOPPED``, the operator submits fresh. If a reconnect finds the run still stopping and it
+settles into ``STOPPED`` while polling, the operator raises instead of returning a result -- the
+task fails and a normal retry resubmits. Either way, a self-inflicted stop never gets silently
+reported as a false success.
+
+This protection also applies when ``wait_for_completion=False`` -- even though that task attempt
+never polls at all, a retry after a successful submission still reconnects rather than
+resubmitting, since the run id is persisted immediately after submission regardless of whether the
+task waits for it to finish.
+
+Durable execution requires Airflow 3.3 or newer for the task state store lookup above. On earlier
+Airflow versions, or if a prior run was never recorded to the task state store, ``durable=True``
+still recovers a prior run via an older mechanism: the operator checks XCom for a cached run id
+first, then falls back to scanning the job's run history for a run tagged with this task
+instance's identity, and reconnects if it finds one that is still active.
+
+Like the persisted state itself, the stored run id isn't deleted automatically, that only happens
+when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is longer than
+``[state_store] default_retention_days`` (30 days by default) and cleanup runs in between, the run
+id won't be there for the next retry, and the operator falls back to the XCom/scan mechanism
+above rather than reconnecting via task state store. Avoid running cleanup on a schedule shorter
+than your longest ``retry_delay``.
+
+Clearing a task is treated the same as a retry, which matters specifically for a task whose job
+already succeeded: clearing does not delete the stored run id, so the next attempt reads it back
+and returns immediately without submitting anything to Glue. See
+:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
+``[state_store] clear_on_success`` setting that restores "clearing always resubmits."
+
+To opt out and always start a fresh run on retry, set ``durable=False``:
+
+.. code-block:: python
+
+  glue_job = GlueJobOperator(
+      task_id="glue_job",
+      job_name="my_glue_job",
+      script_location="s3://glue-examples/glue-scripts/sample_aws_glue_job.py",
+      durable=False,
+  )
+
+The task state store lookup above is only used on the synchronous path -- when ``deferrable=True``
+is set, the Triggerer already tracks the run across the wait, so a run id is never persisted there.
+``durable`` still has an effect on retry, though: a retry of a deferrable task resubmits by

Review Comment:
   Replaced "resubmits by default" to "would otherwise resubmit."



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3765651806


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +364,147 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """Record the run id and surface its console link; guarded so one attempt logs it once."""
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:

Review Comment:
   Fixed `_build_script_args` and `submit_job` now skip the tag/scan on Airflow 3.3+ synchronous runs entirely, since task_state_store is the sole reconnect mechanism there. Trades away the narrow crash-before-persist safety net in exchange for never touching argv by default and I accepted that trade since its a rare, self-healing race vs. an immediate breakage risk for strict-parsing scripts.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #71211:
URL: https://github.com/apache/airflow/pull/71211#issuecomment-5276321874

   Wanted to clear up some deprecation rules, and its handled now in: 8895b4c4955919b6eef8135495a580e7583f3760


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3756133975


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +366,150 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable and not self.deferrable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        # The --airflow_task_uuid tag is still written on every nondeferred attempt,
+        # including the first: a later retry's scan can only find it if it was there from the
+        # start. The lookup itself is skipped on the first attempt, since nothing tagged with this
+        # task's UUID could possibly exist yet -- paying for a full run-history scan there would
+        # be pure waste.
+        if self.durable and not self.deferrable and context["ti"].try_number > 1:
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])
+            if existing_job_run_id:
+                self._set_job_run_id(context, existing_job_run_id)
+                return existing_job_run_id
+        self.log.info(
+            "Initializing AWS Glue Job: %s. Wait for completion: %s",
+            self.job_name,
+            self.wait_for_completion,
+        )
+        glue_job_run = self.hook.initialize_job(script_args, self.run_job_kwargs)
+        # Set immediately (before any polling) so on_kill can stop the run even if the worker dies
+        # before poll_until_complete runs.
+        self._set_job_run_id(context, glue_job_run["JobRunId"])
+        # Pushed unconditionally, matching pre-durable-execution behavior: downstream tasks read
+        # this key directly, and it's also the only writer that makes _find_previous_job_run's
+        # XCom tier reachable on the next retry.
+        context["ti"].xcom_push(key="glue_job_run_id", value=glue_job_run["JobRunId"])
+        return glue_job_run["JobRunId"]
+
+    def get_job_status(self, external_id: JsonValue, context: Context) -> str:
+        """Query the raw job run state; a run id Glue no longer knows about degrades to NOT_FOUND."""
+        job_run_id = cast("str", external_id)
+        # This is the first place a reconnecting attempt learns the run id.
+        self._set_job_run_id(context, job_run_id)
+        try:
+            return self.hook.get_job_state(self.job_name, job_run_id)
+        except ClientError as e:
+            if e.response["Error"]["Code"] == "EntityNotFoundException":
+                return NOT_FOUND_STATE
+            raise
+
+    def is_job_active(self, status: str) -> bool:
+        return status not in (*JOB_RUN_TERMINAL_STATES, NOT_FOUND_STATE)

Review Comment:
   Thanks, detailed review. I confirmed by reproducing your exact scenario. Went with option 2 - polling now checks the state the hook's poll loop actually finished on, and raises (new `GlueJobRunStoppedError`) if it's 
   not a real success. 
   
   STOPPING still correctly counts as active - nothing changes there, this only intervenes at the moment the hook's own finished_states disagrees with ours. Added a test that reproduces your log sequence exactly (STOPPING -> STOPPED) and confirmed it fails without the fix.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3756741177


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +366,150 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable and not self.deferrable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        # The --airflow_task_uuid tag is still written on every nondeferred attempt,
+        # including the first: a later retry's scan can only find it if it was there from the
+        # start. The lookup itself is skipped on the first attempt, since nothing tagged with this
+        # task's UUID could possibly exist yet -- paying for a full run-history scan there would
+        # be pure waste.
+        if self.durable and not self.deferrable and context["ti"].try_number > 1:
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])
+            if existing_job_run_id:
+                self._set_job_run_id(context, existing_job_run_id)
+                return existing_job_run_id
+        self.log.info(
+            "Initializing AWS Glue Job: %s. Wait for completion: %s",
+            self.job_name,
+            self.wait_for_completion,
+        )
+        glue_job_run = self.hook.initialize_job(script_args, self.run_job_kwargs)
+        # Set immediately (before any polling) so on_kill can stop the run even if the worker dies
+        # before poll_until_complete runs.
+        self._set_job_run_id(context, glue_job_run["JobRunId"])
+        # Pushed unconditionally, matching pre-durable-execution behavior: downstream tasks read
+        # this key directly, and it's also the only writer that makes _find_previous_job_run's
+        # XCom tier reachable on the next retry.
+        context["ti"].xcom_push(key="glue_job_run_id", value=glue_job_run["JobRunId"])
+        return glue_job_run["JobRunId"]
+
+    def get_job_status(self, external_id: JsonValue, context: Context) -> str:
+        """Query the raw job run state; a run id Glue no longer knows about degrades to NOT_FOUND."""
+        job_run_id = cast("str", external_id)
+        # This is the first place a reconnecting attempt learns the run id.
+        self._set_job_run_id(context, job_run_id)
+        try:
+            return self.hook.get_job_state(self.job_name, job_run_id)
+        except ClientError as e:
+            if e.response["Error"]["Code"] == "EntityNotFoundException":
+                return NOT_FOUND_STATE
+            raise
+
+    def is_job_active(self, status: str) -> bool:
+        return status not in (*JOB_RUN_TERMINAL_STATES, NOT_FOUND_STATE)

Review Comment:
   Handled in 118dacfd80



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3764958331


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +364,147 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """Record the run id and surface its console link; guarded so one attempt logs it once."""
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first, then falls back to a task-UUID scan. The XCom tier
+        only works on Airflow 2.11-3.2; Airflow 3.3+ clears task XComs before every non-deferral
+        attempt, so it always misses there and the scan runs every time.
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):

Review Comment:
   Fixed, both reconnect checks in `_find_previous_job_run` now call `self.is_job_active(state)` instead of the old hardcoded values, so WAITING and STOPPING are treated as reconnectable there too, matching `is_job_active` and the docs. Added parametrized coverage for both states on both tiers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #71211:
URL: https://github.com/apache/airflow/pull/71211#issuecomment-5277916466

   Thanks for the review @kaxil! Merging this one in.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh closed pull request #71211: Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator`
URL: https://github.com/apache/airflow/pull/71211


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3765553848


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +364,147 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """Record the run id and surface its console link; guarded so one attempt logs it once."""
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first, then falls back to a task-UUID scan. The XCom tier

Review Comment:
   Reworded both the docstring and glue docs to say "every Airflow 3 release" clears XComs, not just 3.3+. Leaving the XCom tier in place for now (not dropping it), and leaving those two tests as-is since they're still valid for 2.11, just not reachable on 3.x which the reworded docs now say plainly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3765064600


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +364,147 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """Record the run id and surface its console link; guarded so one attempt logs it once."""
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first, then falls back to a task-UUID scan. The XCom tier
+        only works on Airflow 2.11-3.2; Airflow 3.3+ clears task XComs before every non-deferral
+        attempt, so it always misses there and the scan runs every time.
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def _has_stored_external_id(self, context: Context) -> bool:
+        task_state_store = context.get("task_state_store")
+        return task_state_store is not None and task_state_store.get(self.external_id_key) is not None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        # Scan only when there's nothing else to go on: first attempt, no store, or a store that
+        # never recorded this key. A stored id (even terminal) means the caller already decided.
+        if self.durable and context["ti"].try_number > 1 and not self._has_stored_external_id(context):

Review Comment:
   I'll track it here: https://github.com/apache/airflow/issues/71485



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3765774331


##########
providers/amazon/src/airflow/providers/amazon/aws/exceptions.py:
##########
@@ -78,3 +78,7 @@ class NeptuneImportTaskCancellationFailedError(AirflowException):
 
 class NeptuneImportTaskFailedError(AirflowException):
     """Raised when a Neptune Analytics import task fails to complete successfully."""
+
+
+class GlueJobRunStoppedError(AirflowException):
+    """Raised when a reconnected Glue job run finishes in a state that is not a real success."""

Review Comment:
   Dropped "reconnected" from the docstring.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3747204778


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +363,146 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        if self.durable:
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])
+            if existing_job_run_id:
+                self._set_job_run_id(context, existing_job_run_id)
+                return existing_job_run_id
+        self.log.info(
+            "Initializing AWS Glue Job: %s. Wait for completion: %s",
+            self.job_name,
+            self.wait_for_completion,
+        )
+        glue_job_run = self.hook.initialize_job(script_args, self.run_job_kwargs)
+        # Set immediately (before any polling) so on_kill can stop the run even if the worker dies
+        # before poll_until_complete runs.
+        self._set_job_run_id(context, glue_job_run["JobRunId"])
+        return glue_job_run["JobRunId"]
+
+    def get_job_status(self, external_id: JsonValue, context: Context) -> str:
+        """Query the raw job run state; a run id Glue no longer knows about degrades to NOT_FOUND."""
+        job_run_id = cast("str", external_id)
+        # This is the first place a reconnecting attempt learns the run id.
+        self._set_job_run_id(context, job_run_id)
+        try:
+            return self.hook.get_job_state(self.job_name, job_run_id)
+        except ClientError as e:
+            if e.response["Error"]["Code"] == "EntityNotFoundException":
+                return "NOT_FOUND"

Review Comment:
   Woops, handled.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3747045390


##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -139,6 +139,70 @@ To submit a new AWS Glue job you can use :class:`~airflow.providers.amazon.aws.o
   The same AWS IAM role used for the crawler can be used here as well, but it will need
   policies to provide access to the output location for result data.
 
+Durable execution
+==================
+
+``GlueJobOperator`` submits a job run and then polls it to completion on the worker. By default
+the operator runs in a *durable* mode that makes this crash-safe: the Glue job run id is
+persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>` before
+polling begins, so if the worker crashes or is preempted and the task is retried, the operator
+reconnects to the run that is already executing in Glue instead of starting a new one.
+
+This matters more for Glue because a Glue job's ``concurrent_run_limit`` defaults to ``1``, so
+submitting a second run while the first is still active does not create a harmless duplicate, it
+fails outright with ``ConcurrentRunsExceededException`` and the task keeps retrying against a run
+it can never see. Durable execution turns that retry into a normal reconnect.
+
+On retry the operator checks the prior run's state:
+
+* if it is still starting, running, waiting for capacity, or being stopped, the operator
+  reconnects and continues polling
+* if it already succeeded, or was stopped outside Airflow, the operator returns immediately
+  without resubmitting

Review Comment:
   Fixed. Updated the docs because I think clearing task should reconnect to an existing job if its present. Documented the escape hatch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3772474228


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -158,6 +203,16 @@ def __init__(
         ),
         **kwargs,
     ):
+        if resume_glue_job_on_retry is not None:
+            # Kept as a real named parameter (not **kwargs) so `default_args` still applies correctly.

Review Comment:
   Wanted to provide an update on this one. I thought more abnout this and went a step further than the original fix: 
   Below Airflow 3.3, durable's value is now fully inert, not just correctly forwarded via default_args, but ignored outright even if explicitly set, with a warning. 
   Below 3.3, only `resume_glue_job_on_retry` controls behavior there, exactly as it did before this port existed. 
   
   Wanted to flag this since it's a further behavior change in the same area you had already reviewed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3747151275


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +363,146 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        if self.durable:
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])

Review Comment:
   Added the`try_number > 1` guard, you are right that it matters now that comment 3 restored the xcom write, and the docs already claimed this was fallback only behavior. The `--airflow_task_uuid` tag itself still gets written on every attempt including the first (a later retry's scan needs it to have been there from the start), only the lookup is skipped.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3756344194


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +366,150 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable and not self.deferrable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        # The --airflow_task_uuid tag is still written on every nondeferred attempt,
+        # including the first: a later retry's scan can only find it if it was there from the
+        # start. The lookup itself is skipped on the first attempt, since nothing tagged with this
+        # task's UUID could possibly exist yet -- paying for a full run-history scan there would
+        # be pure waste.
+        if self.durable and not self.deferrable and context["ti"].try_number > 1:

Review Comment:
   Fixed. submit_job now checks state store directly. If it already has a value for the run-id key (even a terminal one), the scan is skipped entirely, since your own check already ruled out reconnecting. It only runs when there's genuinely nothing recorded: first attempt, no store (pre-3.3), or a store that's never seen this key.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3747199576


##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -139,6 +139,70 @@ To submit a new AWS Glue job you can use :class:`~airflow.providers.amazon.aws.o
   The same AWS IAM role used for the crawler can be used here as well, but it will need
   policies to provide access to the output location for result data.
 
+Durable execution
+==================
+
+``GlueJobOperator`` submits a job run and then polls it to completion on the worker. By default
+the operator runs in a *durable* mode that makes this crash-safe: the Glue job run id is
+persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>` before
+polling begins, so if the worker crashes or is preempted and the task is retried, the operator
+reconnects to the run that is already executing in Glue instead of starting a new one.
+
+This matters more for Glue because a Glue job's ``concurrent_run_limit`` defaults to ``1``, so
+submitting a second run while the first is still active does not create a harmless duplicate, it
+fails outright with ``ConcurrentRunsExceededException`` and the task keeps retrying against a run
+it can never see. Durable execution turns that retry into a normal reconnect.
+
+On retry the operator checks the prior run's state:
+
+* if it is still starting, running, waiting for capacity, or being stopped, the operator
+  reconnects and continues polling
+* if it already succeeded, or was stopped outside Airflow, the operator returns immediately
+  without resubmitting
+* if it failed terminally, or its id has expired and is no longer found, the operator submits the
+  job fresh
+
+A run that was stopped outside Airflow (for example, cancelled manually in the AWS console) is
+treated as a success rather than resubmitted, since the work is genuinely finished, just not the
+way the task expected - the operator logs a warning when this happens.
+
+This protection also applies when ``wait_for_completion=False`` -- even though that task attempt
+never polls at all, a retry after a successful submission still reconnects rather than
+resubmitting, since the run id is persisted immediately after submission regardless of whether the
+task waits for it to finish.
+
+Durable execution requires Airflow 3.3 or newer for the task state store lookup above. On earlier
+Airflow versions, or if the task state store is unavailable at runtime, ``durable=True`` still
+recovers a prior run, just via an older mechanism: the operator checks XCom for a cached run id
+first, then falls back to scanning the job's run history for a run tagged with this task
+instance's identity, and reconnects if it finds one that is still active.
+
+Like the persisted state itself, the stored run id isn't deleted automatically, that only happens
+when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is longer than
+``[state_store] default_retention_days`` (30 days by default) and cleanup runs in between, the run
+id won't be there for the next retry, and the operator falls back to the XCom/scan mechanism
+above rather than reconnecting via task state store. Avoid running cleanup on a schedule shorter
+than your longest ``retry_delay``.
+
+To opt out and always start a fresh run on retry, set ``durable=False``:
+
+.. code-block:: python
+
+  glue_job = GlueJobOperator(
+      task_id="glue_job",
+      job_name="my_glue_job",
+      script_location="s3://glue-examples/glue-scripts/sample_aws_glue_job.py",
+      durable=False,
+  )
+
+Durable execution applies to the synchronous path. When ``deferrable=True`` is set, the Triggerer
+already tracks the run across the wait, so deferrable mode takes precedence and ``durable`` has no

Review Comment:
   Thanks, gated stuff on deferrable now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3764995429


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +364,147 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """Record the run id and surface its console link; guarded so one attempt logs it once."""
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first, then falls back to a task-UUID scan. The XCom tier
+        only works on Airflow 2.11-3.2; Airflow 3.3+ clears task XComs before every non-deferral
+        attempt, so it always misses there and the scan runs every time.
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def _has_stored_external_id(self, context: Context) -> bool:
+        task_state_store = context.get("task_state_store")
+        return task_state_store is not None and task_state_store.get(self.external_id_key) is not None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        # Scan only when there's nothing else to go on: first attempt, no store, or a store that
+        # never recorded this key. A stored id (even terminal) means the caller already decided.
+        if self.durable and context["ti"].try_number > 1 and not self._has_stored_external_id(context):
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])
+            if existing_job_run_id:
+                self._set_job_run_id(context, existing_job_run_id)
+                return existing_job_run_id
+        self.log.info(
+            "Initializing AWS Glue Job: %s. Wait for completion: %s",
+            self.job_name,
+            self.wait_for_completion,
+        )
+        # A prior get_job_status call may have set this to a stale, terminal run id while checking
+        # whether to reconnect. Clear it so on_kill has nothing to act on if initialize_job raises.
+        self._job_run_id = None
+        glue_job_run = self.hook.initialize_job(script_args, self.run_job_kwargs)
+        # Set before polling so on_kill can stop the run even if the worker dies immediately after.
+        self._set_job_run_id(context, glue_job_run["JobRunId"])
+        # Downstream tasks read this key directly; it's also what feeds the XCom tier above.
+        context["ti"].xcom_push(key="glue_job_run_id", value=glue_job_run["JobRunId"])

Review Comment:
   Moved the push into `_set_job_run_id`, which already runs on every path (fresh submit, reconnect, alreadysucceeded, bootstrap scan). Added test coverage on the reconnect and already-succeeded paths.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3765738405


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -35,17 +38,52 @@
 )
 from airflow.providers.amazon.aws.utils import validate_execute_complete_event
 from airflow.providers.amazon.aws.utils.mixins import aws_template_fields
+from airflow.providers.amazon.version_compat import AIRFLOW_V_3_3_PLUS
 from airflow.providers.common.compat.openlineage.utils.spark import (
     inject_parent_job_information_into_glue_arguments,
     inject_transport_information_into_glue_arguments,
 )
 from airflow.providers.common.compat.sdk import AirflowException, conf
 
+# ResumableJobMixin only exists on Airflow 3; this provider still targets >=2.11. Drop this
+# fallback once the provider's minimum Airflow version is >=3.0.

Review Comment:
   Good catch, fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3747078190


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +363,146 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        if self.durable:
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])
+            if existing_job_run_id:
+                self._set_job_run_id(context, existing_job_run_id)
+                return existing_job_run_id
+        self.log.info(
+            "Initializing AWS Glue Job: %s. Wait for completion: %s",
+            self.job_name,
+            self.wait_for_completion,
+        )
+        glue_job_run = self.hook.initialize_job(script_args, self.run_job_kwargs)
+        # Set immediately (before any polling) so on_kill can stop the run even if the worker dies
+        # before poll_until_complete runs.
+        self._set_job_run_id(context, glue_job_run["JobRunId"])

Review Comment:
   restored the unconditional xcom_push after a fresh submit (matches the pre-durable-execution code exactly: it ran outside the resume_glue_job_on_retry check, so it should run regardless of durable too). 
   
   Fixes both consequences: downstream `xcom_pull(key="glue_job_run_id")` works again, and `_find_previous_job_run`'s xcom tier is reachable on the next retry. Added back the assertion that used to guard this `(assert_any_call(key="glue_job_run_id", ...))`, and confirmed it actually fails without the fix before adding it.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3765784984


##########
providers/amazon/tests/unit/amazon/aws/operators/test_glue.py:
##########
@@ -163,6 +171,62 @@ def test_execute_deferrable(self, _, mock_initialize_job):
         assert defer.value.trigger.attempts == 75
         assert defer.value.trigger.aws_conn_id == "aws_default"
 
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_deferrable_first_attempt_injects_task_uuid_but_skips_scan(

Review Comment:
   Added the missing `get_job_runs.assert_not_called()` assertion, same pattern as the sibling test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3755982014


##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -139,6 +139,79 @@ To submit a new AWS Glue job you can use :class:`~airflow.providers.amazon.aws.o
   The same AWS IAM role used for the crawler can be used here as well, but it will need
   policies to provide access to the output location for result data.
 
+Durable execution
+==================
+
+``GlueJobOperator`` submits a job run and then polls it to completion on the worker. By default
+the operator runs in a *durable* mode that makes this crash-safe: the Glue job run id is
+persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>` before
+polling begins, so if the worker crashes or is preempted and the task is retried, the operator
+reconnects to the run that is already executing in Glue instead of starting a new one.
+
+This matters more for Glue because a Glue job's ``concurrent_run_limit`` defaults to ``1``, so
+submitting a second run while the first is still active does not create a harmless duplicate, it
+fails outright with ``ConcurrentRunsExceededException`` and the task keeps retrying against a run
+it can never see. Durable execution turns that retry into a normal reconnect.
+
+On retry the operator checks the prior run's state:
+
+* if it is still starting, running, waiting for capacity, or being stopped, the operator

Review Comment:
   This bullet says a run that is "being stopped" reconnects and continues polling, and the paragraph at 164-170 says a stopped run always resubmits so a self-inflicted stop is never reported as a false success. The code implements this bullet, and reconnecting to a `STOPPING` run ends with `_handle_state` reporting the resulting `STOPPED` as finished, which is exactly what defeats the paragraph below. Whichever way the code fix goes, this bullet needs to move with it.



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +366,150 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable and not self.deferrable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        # The --airflow_task_uuid tag is still written on every nondeferred attempt,
+        # including the first: a later retry's scan can only find it if it was there from the
+        # start. The lookup itself is skipped on the first attempt, since nothing tagged with this
+        # task's UUID could possibly exist yet -- paying for a full run-history scan there would
+        # be pure waste.
+        if self.durable and not self.deferrable and context["ti"].try_number > 1:
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])
+            if existing_job_run_id:
+                self._set_job_run_id(context, existing_job_run_id)
+                return existing_job_run_id
+        self.log.info(
+            "Initializing AWS Glue Job: %s. Wait for completion: %s",
+            self.job_name,
+            self.wait_for_completion,
+        )
+        glue_job_run = self.hook.initialize_job(script_args, self.run_job_kwargs)
+        # Set immediately (before any polling) so on_kill can stop the run even if the worker dies
+        # before poll_until_complete runs.
+        self._set_job_run_id(context, glue_job_run["JobRunId"])
+        # Pushed unconditionally, matching pre-durable-execution behavior: downstream tasks read
+        # this key directly, and it's also the only writer that makes _find_previous_job_run's
+        # XCom tier reachable on the next retry.
+        context["ti"].xcom_push(key="glue_job_run_id", value=glue_job_run["JobRunId"])
+        return glue_job_run["JobRunId"]
+
+    def get_job_status(self, external_id: JsonValue, context: Context) -> str:
+        """Query the raw job run state; a run id Glue no longer knows about degrades to NOT_FOUND."""
+        job_run_id = cast("str", external_id)
+        # This is the first place a reconnecting attempt learns the run id.
+        self._set_job_run_id(context, job_run_id)
+        try:
+            return self.hook.get_job_state(self.job_name, job_run_id)
+        except ClientError as e:
+            if e.response["Error"]["Code"] == "EntityNotFoundException":
+                return NOT_FOUND_STATE
+            raise
+
+    def is_job_active(self, status: str) -> bool:
+        return status not in (*JOB_RUN_TERMINAL_STATES, NOT_FOUND_STATE)

Review Comment:
   `STOPPING` isn't in `JOB_RUN_TERMINAL_STATES`, so it counts as active and a retry reconnects to it. `batch_stop_job_run` is asynchronous, so a run this operator's own `on_kill` just stopped sits in `STOPPING` for a while, and a retry landing in that window reconnects, polls, and `_handle_state`'s `finished_states = ["SUCCEEDED", "STOPPED"]` returns normally, so the task goes green with the ETL half done.
   
   I ran it against the real hook. Serving `STOPPING, STOPPING, STOPPED` gives `Reconnecting to existing job status=STOPPING`, then `Exiting Job jr_old Run State: STOPPED`, `initialize_job` never called, task succeeds. On the merge-base the same scenario resubmits, because the old check was an allow-list (`state in ("RUNNING", "STARTING")`) rather than a deny-list. So this is the same false success the `STOPPED` change was meant to remove, reached through `STOPPING` instead, and it is now on the default path.
   
   Adding `STOPPING` to the terminal tuple trades the silent success for a probably-transient `ConcurrentRunsExceededException`. The narrower fix is to raise in `poll_until_complete` when `job_completion` returns a state outside `JOB_RUN_SUCCESS_STATES`, which also removes the disagreement between that tuple and the hook's `finished_states`. Clearing a running task is the most reachable route into this, since it reschedules immediately rather than waiting out a `retry_delay`.



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +366,150 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable and not self.deferrable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        # The --airflow_task_uuid tag is still written on every nondeferred attempt,
+        # including the first: a later retry's scan can only find it if it was there from the
+        # start. The lookup itself is skipped on the first attempt, since nothing tagged with this
+        # task's UUID could possibly exist yet -- paying for a full run-history scan there would
+        # be pure waste.
+        if self.durable and not self.deferrable and context["ti"].try_number > 1:

Review Comment:
   The `try_number` guard removes the first-attempt cost, but not the case that matters on 3.3+. When the task state store returns a terminal run id, `execute_resumable` logs "Prior job in terminal state, resubmitting fresh" and calls `submit_job`, which runs `_find_previous_job_run` anyway, and since Airflow 3 clears every one of the task instance's XComs before a non-deferral attempt that always takes the scan branch.
   
   With the store returning `FAILED` I get a full walk of the run history at 50 per page before `initialize_job` fires, so a job with a few thousand runs is dozens of `GetJobRuns` calls on every retry after every failure. That is the cost the PR's own "one call vs. a full scan" rationale says durable execution removes, and the docs call this tier a one-time bootstrap. `submit_job` can't see the mixin's decision, so it needs something explicit, like a flag set when `task_state_store.get()` returned an id, leaving the bootstrap lookup live only for the store-empty and pre-upgrade cases.



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +366,150 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable and not self.deferrable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        # The --airflow_task_uuid tag is still written on every nondeferred attempt,
+        # including the first: a later retry's scan can only find it if it was there from the
+        # start. The lookup itself is skipped on the first attempt, since nothing tagged with this
+        # task's UUID could possibly exist yet -- paying for a full run-history scan there would
+        # be pure waste.
+        if self.durable and not self.deferrable and context["ti"].try_number > 1:
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])
+            if existing_job_run_id:
+                self._set_job_run_id(context, existing_job_run_id)
+                return existing_job_run_id
+        self.log.info(
+            "Initializing AWS Glue Job: %s. Wait for completion: %s",
+            self.job_name,
+            self.wait_for_completion,
+        )
+        glue_job_run = self.hook.initialize_job(script_args, self.run_job_kwargs)
+        # Set immediately (before any polling) so on_kill can stop the run even if the worker dies
+        # before poll_until_complete runs.
+        self._set_job_run_id(context, glue_job_run["JobRunId"])
+        # Pushed unconditionally, matching pre-durable-execution behavior: downstream tasks read
+        # this key directly, and it's also the only writer that makes _find_previous_job_run's
+        # XCom tier reachable on the next retry.

Review Comment:
   The second clause isn't true on Airflow 3.x. `xcom_keys_to_clear` is built from a query that selects every key for the task instance with no key filter, and it is skipped only for a deferral, so `glue_job_run_id` is always gone before a retry runs. That is the finding the PR description leads with. The XCom tier is only live on 2.11 through 3.2, so the push is right for the downstream-consumers reason but not for this one. Worth version-qualifying it, otherwise the next reader takes a path that is dead on 3.x for a live recovery tier.



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +366,150 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable and not self.deferrable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        # The --airflow_task_uuid tag is still written on every nondeferred attempt,
+        # including the first: a later retry's scan can only find it if it was there from the
+        # start. The lookup itself is skipped on the first attempt, since nothing tagged with this
+        # task's UUID could possibly exist yet -- paying for a full run-history scan there would
+        # be pure waste.
+        if self.durable and not self.deferrable and context["ti"].try_number > 1:
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])
+            if existing_job_run_id:
+                self._set_job_run_id(context, existing_job_run_id)
+                return existing_job_run_id
+        self.log.info(
+            "Initializing AWS Glue Job: %s. Wait for completion: %s",
+            self.job_name,
+            self.wait_for_completion,
+        )
+        glue_job_run = self.hook.initialize_job(script_args, self.run_job_kwargs)
+        # Set immediately (before any polling) so on_kill can stop the run even if the worker dies
+        # before poll_until_complete runs.
+        self._set_job_run_id(context, glue_job_run["JobRunId"])
+        # Pushed unconditionally, matching pre-durable-execution behavior: downstream tasks read
+        # this key directly, and it's also the only writer that makes _find_previous_job_run's
+        # XCom tier reachable on the next retry.
+        context["ti"].xcom_push(key="glue_job_run_id", value=glue_job_run["JobRunId"])
+        return glue_job_run["JobRunId"]
+
+    def get_job_status(self, external_id: JsonValue, context: Context) -> str:
+        """Query the raw job run state; a run id Glue no longer knows about degrades to NOT_FOUND."""
+        job_run_id = cast("str", external_id)
+        # This is the first place a reconnecting attempt learns the run id.
+        self._set_job_run_id(context, job_run_id)

Review Comment:
   On the terminal-resubmit path this persists `GlueJobRunDetailsLink` and logs "You can monitor this Glue Job run at" for the run that is about to be discarded, and then both get rewritten for the new id. My probe log shows `.../run/jr_old` immediately followed by `.../run/jr_new` within one attempt. If `initialize_job` then raises, which `ConcurrentRunsExceededException` makes realistic here, `self._job_run_id` is left pointing at the stale terminal run, and that is what `on_kill` would try to stop.



##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -139,6 +139,79 @@ To submit a new AWS Glue job you can use :class:`~airflow.providers.amazon.aws.o
   The same AWS IAM role used for the crawler can be used here as well, but it will need
   policies to provide access to the output location for result data.
 
+Durable execution
+==================
+
+``GlueJobOperator`` submits a job run and then polls it to completion on the worker. By default
+the operator runs in a *durable* mode that makes this crash-safe: the Glue job run id is
+persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>` before
+polling begins, so if the worker crashes or is preempted and the task is retried, the operator
+reconnects to the run that is already executing in Glue instead of starting a new one.
+
+This matters more for Glue because a Glue job's ``concurrent_run_limit`` defaults to ``1``, so
+submitting a second run while the first is still active does not create a harmless duplicate, it
+fails outright with ``ConcurrentRunsExceededException`` and the task keeps retrying against a run
+it can never see. Durable execution turns that retry into a normal reconnect.
+
+On retry the operator checks the prior run's state:
+
+* if it is still starting, running, waiting for capacity, or being stopped, the operator
+  reconnects and continues polling
+* if it already succeeded, the operator returns immediately without resubmitting
+* if it stopped, failed terminally, or its id has expired and is no longer found, the operator
+  submits the job fresh
+
+A stopped run resubmits rather than being treated as a success. Glue's API has no way to tell a
+run that was cancelled manually (for example, in the AWS console) apart from one this operator's
+own :meth:`~airflow.providers.amazon.aws.operators.glue.GlueJobOperator.on_kill` stopped, which
+happens whenever ``stop_job_run_on_kill=True`` and the task is killed -- on SIGTERM, on
+``execution_timeout``, or when the task is cleared while running. Since the two cases can't be
+told apart, a stopped run always resubmits, so a self-inflicted stop never gets silently reported
+as a false success.
+
+This protection also applies when ``wait_for_completion=False`` -- even though that task attempt
+never polls at all, a retry after a successful submission still reconnects rather than
+resubmitting, since the run id is persisted immediately after submission regardless of whether the
+task waits for it to finish.
+
+Durable execution requires Airflow 3.3 or newer for the task state store lookup above. On earlier
+Airflow versions, or if the task state store is unavailable at runtime, ``durable=True`` still
+recovers a prior run, just via an older mechanism: the operator checks XCom for a cached run id
+first, then falls back to scanning the job's run history for a run tagged with this task
+instance's identity, and reconnects if it finds one that is still active.
+
+Like the persisted state itself, the stored run id isn't deleted automatically, that only happens
+when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is longer than
+``[state_store] default_retention_days`` (30 days by default) and cleanup runs in between, the run
+id won't be there for the next retry, and the operator falls back to the XCom/scan mechanism
+above rather than reconnecting via task state store. Avoid running cleanup on a schedule shorter
+than your longest ``retry_delay``.
+
+Clearing a task is treated the same as a retry, which matters specifically for a task whose job
+already succeeded: clearing does not delete the stored run id, so the next attempt reads it back
+and returns immediately without submitting anything to Glue. See
+:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the

Review Comment:
   `core-concepts/resumable-tasks` never mentions `clear_on_success`, or clearing at all. The setting is documented in `core-concepts/task-state-store` under "Automatic cleanup (``clear_on_success``)", which this page already links at line 147, so the link just needs repointing.



##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -139,6 +139,79 @@ To submit a new AWS Glue job you can use :class:`~airflow.providers.amazon.aws.o
   The same AWS IAM role used for the crawler can be used here as well, but it will need
   policies to provide access to the output location for result data.
 
+Durable execution
+==================
+
+``GlueJobOperator`` submits a job run and then polls it to completion on the worker. By default
+the operator runs in a *durable* mode that makes this crash-safe: the Glue job run id is
+persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>` before
+polling begins, so if the worker crashes or is preempted and the task is retried, the operator
+reconnects to the run that is already executing in Glue instead of starting a new one.
+
+This matters more for Glue because a Glue job's ``concurrent_run_limit`` defaults to ``1``, so
+submitting a second run while the first is still active does not create a harmless duplicate, it
+fails outright with ``ConcurrentRunsExceededException`` and the task keeps retrying against a run
+it can never see. Durable execution turns that retry into a normal reconnect.
+
+On retry the operator checks the prior run's state:
+
+* if it is still starting, running, waiting for capacity, or being stopped, the operator
+  reconnects and continues polling
+* if it already succeeded, the operator returns immediately without resubmitting
+* if it stopped, failed terminally, or its id has expired and is no longer found, the operator
+  submits the job fresh
+
+A stopped run resubmits rather than being treated as a success. Glue's API has no way to tell a
+run that was cancelled manually (for example, in the AWS console) apart from one this operator's
+own :meth:`~airflow.providers.amazon.aws.operators.glue.GlueJobOperator.on_kill` stopped, which
+happens whenever ``stop_job_run_on_kill=True`` and the task is killed -- on SIGTERM, on
+``execution_timeout``, or when the task is cleared while running. Since the two cases can't be
+told apart, a stopped run always resubmits, so a self-inflicted stop never gets silently reported
+as a false success.
+
+This protection also applies when ``wait_for_completion=False`` -- even though that task attempt
+never polls at all, a retry after a successful submission still reconnects rather than
+resubmitting, since the run id is persisted immediately after submission regardless of whether the
+task waits for it to finish.
+
+Durable execution requires Airflow 3.3 or newer for the task state store lookup above. On earlier
+Airflow versions, or if the task state store is unavailable at runtime, ``durable=True`` still

Review Comment:
   "unavailable" is doing double duty in this sentence, and only one of its readings holds.
   
   Accessor absent from the context (pre-3.3): `execute_resumable` warns and falls through to `submit_job`, so the XCom/scan tier runs. Key simply not stored: the accessor gets `TASK_STORE_NOT_FOUND` back, returns its `default`, and the same fallthrough happens. Both of those match the promise.
   
   Store present but erroring does not. `TaskStateStoreAccessor.get` does IPC to the supervisor, and `_extract_get_response` raises `AirflowRuntimeError` on any error response that isn't `TASK_STORE_NOT_FOUND` (`task-sdk/src/airflow/sdk/execution_time/context.py:586-587`). There is no `except` anywhere in `resumablejobmixin.py`, so that propagates out of `execute_resumable` before `submit_job` is reached, and the XCom/scan tier cannot run, on that attempt or on any retry hitting the same error. That is the case this sentence is specifically promising it covers.
   
   The raise is deliberate in the shared helper and you have scoped task-sdk changes out, so narrowing the wording to the absent and not-stored cases looks like the right fix here.



##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -139,6 +139,79 @@ To submit a new AWS Glue job you can use :class:`~airflow.providers.amazon.aws.o
   The same AWS IAM role used for the crawler can be used here as well, but it will need
   policies to provide access to the output location for result data.
 
+Durable execution
+==================
+
+``GlueJobOperator`` submits a job run and then polls it to completion on the worker. By default
+the operator runs in a *durable* mode that makes this crash-safe: the Glue job run id is
+persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>` before
+polling begins, so if the worker crashes or is preempted and the task is retried, the operator
+reconnects to the run that is already executing in Glue instead of starting a new one.
+
+This matters more for Glue because a Glue job's ``concurrent_run_limit`` defaults to ``1``, so
+submitting a second run while the first is still active does not create a harmless duplicate, it
+fails outright with ``ConcurrentRunsExceededException`` and the task keeps retrying against a run
+it can never see. Durable execution turns that retry into a normal reconnect.
+
+On retry the operator checks the prior run's state:
+
+* if it is still starting, running, waiting for capacity, or being stopped, the operator
+  reconnects and continues polling
+* if it already succeeded, the operator returns immediately without resubmitting
+* if it stopped, failed terminally, or its id has expired and is no longer found, the operator
+  submits the job fresh
+
+A stopped run resubmits rather than being treated as a success. Glue's API has no way to tell a
+run that was cancelled manually (for example, in the AWS console) apart from one this operator's
+own :meth:`~airflow.providers.amazon.aws.operators.glue.GlueJobOperator.on_kill` stopped, which
+happens whenever ``stop_job_run_on_kill=True`` and the task is killed -- on SIGTERM, on
+``execution_timeout``, or when the task is cleared while running. Since the two cases can't be
+told apart, a stopped run always resubmits, so a self-inflicted stop never gets silently reported
+as a false success.
+
+This protection also applies when ``wait_for_completion=False`` -- even though that task attempt
+never polls at all, a retry after a successful submission still reconnects rather than
+resubmitting, since the run id is persisted immediately after submission regardless of whether the
+task waits for it to finish.
+
+Durable execution requires Airflow 3.3 or newer for the task state store lookup above. On earlier
+Airflow versions, or if the task state store is unavailable at runtime, ``durable=True`` still
+recovers a prior run, just via an older mechanism: the operator checks XCom for a cached run id
+first, then falls back to scanning the job's run history for a run tagged with this task
+instance's identity, and reconnects if it finds one that is still active.
+
+Like the persisted state itself, the stored run id isn't deleted automatically, that only happens
+when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is longer than
+``[state_store] default_retention_days`` (30 days by default) and cleanup runs in between, the run
+id won't be there for the next retry, and the operator falls back to the XCom/scan mechanism
+above rather than reconnecting via task state store. Avoid running cleanup on a schedule shorter
+than your longest ``retry_delay``.
+
+Clearing a task is treated the same as a retry, which matters specifically for a task whose job
+already succeeded: clearing does not delete the stored run id, so the next attempt reads it back
+and returns immediately without submitting anything to Glue. See
+:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
+``[state_store] clear_on_success`` setting that restores "clearing always resubmits."
+
+To opt out and always start a fresh run on retry, set ``durable=False``:
+
+.. code-block:: python
+
+  glue_job = GlueJobOperator(
+      task_id="glue_job",
+      job_name="my_glue_job",
+      script_location="s3://glue-examples/glue-scripts/sample_aws_glue_job.py",
+      durable=False,
+  )
+
+Durable execution applies to the synchronous path. When ``deferrable=True`` is set, the Triggerer
+already tracks the run across the wait, so deferrable mode takes precedence and ``durable`` has no

Review Comment:
   Gating on `not self.deferrable` was my suggestion, but it does give up something main had. There the `resume_glue_job_on_retry` block ran before `self.defer(...)`, so a retry of a deferrable task scanned by task UUID and reattached to the still-running run instead of resubmitting. With the gate, `deferrable=True` always resubmits, which with `concurrent_run_limit` defaulting to 1 is the `ConcurrentRunsExceededException` loop this PR's own rationale describes. Anyone who set `resume_glue_job_on_retry=True` alongside `deferrable=True` loses that, so it is worth saying so here rather than only "durable has no effect", since the deprecation note below implies the value just maps across.



##########
providers/amazon/tests/unit/amazon/aws/operators/test_glue.py:
##########
@@ -781,6 +784,295 @@ def test_inject_parent_job_info_with_resume_on_retry(
         assert GlueJobOperator.TASK_UUID_ARG in call_args
 
 
+class TestGlueJobOperatorDeprecation:
+    @pytest.mark.parametrize("resume_value", [True, False])
+    def test_warns_on_3_3_plus_and_maps_to_durable(self, resume_value):
+        with mock.patch("airflow.providers.amazon.aws.operators.glue.AIRFLOW_V_3_3_PLUS", True):
+            with pytest.warns(AirflowProviderDeprecationWarning, match="resume_glue_job_on_retry"):
+                glue = GlueJobOperator(
+                    task_id=TASK_ID, job_name=JOB_NAME, resume_glue_job_on_retry=resume_value
+                )
+        assert glue.durable is resume_value
+
+    def test_silent_below_3_3(self):
+        with mock.patch("airflow.providers.amazon.aws.operators.glue.AIRFLOW_V_3_3_PLUS", False):
+            with warnings.catch_warnings(record=True) as caught:
+                warnings.simplefilter("always")
+                glue = GlueJobOperator(task_id=TASK_ID, job_name=JOB_NAME, resume_glue_job_on_retry=True)
+            deprecation_warnings = [
+                w for w in caught if issubclass(w.category, AirflowProviderDeprecationWarning)
+            ]
+        assert deprecation_warnings == []
+        assert glue.durable is True
+
+    def test_both_flags_passed_durable_wins(self):
+        with mock.patch("airflow.providers.amazon.aws.operators.glue.AIRFLOW_V_3_3_PLUS", True):
+            with pytest.warns(AirflowProviderDeprecationWarning):
+                glue = GlueJobOperator(
+                    task_id=TASK_ID,
+                    job_name=JOB_NAME,
+                    durable=True,
+                    resume_glue_job_on_retry=False,
+                )
+        assert glue.durable is True
+
+    @pytest.mark.skipif(
+        not AIRFLOW_V_3_3_PLUS,
+        reason="The <3.3 compat stub's __init__ isn't decorated with BaseOperatorMeta._apply_defaults, "
+        "so default_args injection for durable only works on the real ResumableJobMixin.",
+    )
+    def test_default_args_durable_reaches_operator(self):
+        with DAG(
+            dag_id="test_glue_durable_default_args",
+            schedule=None,
+            start_date=datetime(2024, 1, 1),
+            default_args={"durable": False},
+        ):
+            glue = GlueJobOperator(task_id=TASK_ID, job_name=JOB_NAME)
+        assert glue.durable is False
+
+
+class FakeTaskStateStore:
+    """In-memory task state store for tests."""
+
+    def __init__(self, stored: dict[str, str] | None = None):
+        self._store: dict[str, str] = dict(stored or {})
+
+    def get(self, key: str) -> str | None:
+        return self._store.get(key)
+
+    def set(self, key: str, value: str) -> None:
+        self._store[key] = value
+
+
+@pytest.mark.skipif(
+    not AIRFLOW_V_3_3_PLUS,
+    reason="ResumableJobMixin reconnect requires task_state_store, available in Airflow 3.3+",
+)
+class TestGlueJobOperatorDurableExecution:
+    def _build(self, **kwargs):
+        return GlueJobOperator(task_id=TASK_ID, job_name=JOB_NAME, **kwargs)
+
+    def _stub_empty_scan(self, glue):
+        # submit_job scans for a task UUID tagged run whenever durable is set, regardless of why it
+        # was called. Stub it to return no matches so tests don't depend on that fallback mechanism.
+        glue.hook.conn = mock.MagicMock()
+        glue.hook.conn.get_job_runs.return_value = {"JobRuns": []}
+
+    def _context(self, store=None, try_number=2):
+        ti = mock.MagicMock()
+        ti.try_number = try_number
+        ti.xcom_pull.return_value = None
+        ctx = {"ti": ti}
+        if store is not None:
+            ctx["task_state_store"] = store
+        return ctx
+
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_fresh_submit_persists_before_polling(
+        self, mock_get_conn, mock_initialize_job, mock_job_completion
+    ):
+        glue = self._build(durable=True)
+        self._stub_empty_scan(glue)
+        mock_initialize_job.return_value = {"JobRunId": "jr_new"}
+        store = FakeTaskStateStore()
+        persisted_before_poll = []
+        mock_job_completion.side_effect = lambda *a, **k: (
+            persisted_before_poll.append(store.get("glue_job_run_id")) or {"JobRunState": "SUCCEEDED"}
+        )
+
+        job_run_id = glue.execute(self._context(store))
+
+        assert job_run_id == "jr_new"
+        assert store.get("glue_job_run_id") == "jr_new"
+        assert persisted_before_poll == ["jr_new"]
+        mock_initialize_job.assert_called_once()
+
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "get_job_state")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_reconnect_when_stored_run_is_running(
+        self, mock_get_conn, mock_initialize_job, mock_get_job_state, mock_job_completion
+    ):
+        glue = self._build(durable=True)
+        mock_get_job_state.return_value = "RUNNING"
+        mock_job_completion.return_value = {"JobRunState": "SUCCEEDED"}
+        store = FakeTaskStateStore({"glue_job_run_id": "jr_old"})
+
+        job_run_id = glue.execute(self._context(store))
+
+        assert job_run_id == "jr_old"
+        mock_initialize_job.assert_not_called()
+        mock_job_completion.assert_called_once_with(JOB_NAME, "jr_old", False, 0)
+
+    @pytest.mark.parametrize("status", ["STARTING", "RUNNING", "WAITING", "STOPPING"])
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "get_job_state")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_reconnect_from_every_active_state(
+        self, mock_get_conn, mock_initialize_job, mock_get_job_state, mock_job_completion, status
+    ):
+        glue = self._build(durable=True)
+        mock_get_job_state.return_value = status
+        mock_job_completion.return_value = {"JobRunState": "SUCCEEDED"}
+        store = FakeTaskStateStore({"glue_job_run_id": "jr_old"})
+
+        glue.execute(self._context(store))
+
+        mock_initialize_job.assert_not_called()
+
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "get_job_state")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_already_succeeded_returns_without_resubmit(
+        self, mock_get_conn, mock_initialize_job, mock_get_job_state, mock_job_completion
+    ):
+        glue = self._build(durable=True)
+        mock_get_job_state.return_value = "SUCCEEDED"
+        store = FakeTaskStateStore({"glue_job_run_id": "jr_old"})
+
+        job_run_id = glue.execute(self._context(store))
+
+        assert job_run_id == "jr_old"
+        mock_initialize_job.assert_not_called()
+        mock_job_completion.assert_not_called()
+
+    @pytest.mark.parametrize("status", ["FAILED", "TIMEOUT", "STOPPED"])
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "get_job_state")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_terminal_failure_resubmits_fresh(
+        self, mock_get_conn, mock_initialize_job, mock_get_job_state, mock_job_completion, status
+    ):
+        """STOPPED resubmits like any other terminal state: Glue's API can't tell a console
+        cancellation from a run this operator's own on_kill stopped, so treating it as success
+        would risk silently reporting a self-inflicted stop as done."""
+        glue = self._build(durable=True)
+        self._stub_empty_scan(glue)
+        mock_get_job_state.return_value = status
+        mock_initialize_job.return_value = {"JobRunId": "jr_new"}
+        mock_job_completion.return_value = {"JobRunState": "SUCCEEDED"}
+        store = FakeTaskStateStore({"glue_job_run_id": "jr_old"})
+
+        job_run_id = glue.execute(self._context(store))
+
+        assert job_run_id == "jr_new"
+        assert store.get("glue_job_run_id") == "jr_new"
+        mock_initialize_job.assert_called_once()
+
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "get_job_state")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_not_found_resubmits_fresh(
+        self, mock_get_conn, mock_initialize_job, mock_get_job_state, mock_job_completion
+    ):
+        glue = self._build(durable=True)
+        self._stub_empty_scan(glue)
+        mock_get_job_state.side_effect = ClientError(
+            {"Error": {"Code": "EntityNotFoundException", "Message": "gone"}}, "GetJobRun"
+        )
+        mock_initialize_job.return_value = {"JobRunId": "jr_new"}
+        mock_job_completion.return_value = {"JobRunState": "SUCCEEDED"}
+        store = FakeTaskStateStore({"glue_job_run_id": "jr_old"})
+
+        job_run_id = glue.execute(self._context(store))
+
+        assert job_run_id == "jr_new"
+        mock_initialize_job.assert_called_once()
+
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_durable_false_never_touches_store(self, mock_get_conn, mock_initialize_job, mock_job_completion):
+        glue = self._build(durable=False)
+        mock_initialize_job.return_value = {"JobRunId": "jr_new"}
+        mock_job_completion.return_value = {"JobRunState": "SUCCEEDED"}
+        store = FakeTaskStateStore({"glue_job_run_id": "jr_old"})
+
+        job_run_id = glue.execute(self._context(store))
+
+        assert job_run_id == "jr_new"
+        assert store.get("glue_job_run_id") == "jr_old", "store must be left untouched"
+        mock_initialize_job.assert_called_once()
+
+    @mock.patch.object(GlueJobHook, "conn", new_callable=mock.PropertyMock)
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_first_attempt_skips_the_retry_lookup_entirely(
+        self, mock_get_conn, mock_initialize_job, mock_job_completion, mock_conn
+    ):
+        glue = self._build(durable=True)
+        mock_initialize_job.return_value = {"JobRunId": "jr_new"}
+        mock_job_completion.return_value = {"JobRunState": "SUCCEEDED"}
+        store = FakeTaskStateStore()
+
+        job_run_id = glue.execute(self._context(store, try_number=1))
+
+        assert job_run_id == "jr_new"
+        mock_initialize_job.assert_called_once()
+        mock_conn.return_value.get_job_run.assert_not_called()
+        mock_conn.return_value.get_job_runs.assert_not_called()

Review Comment:
   The invariant the comment in `submit_job` rests on, that the tag is still written on attempt 1 so a later retry's scan can find it, isn't asserted anywhere. One more line here, `assert GlueJobOperator.TASK_UUID_ARG in mock_initialize_job.call_args[0][0]`, would stop a later refactor from moving the injection under the `try_number` guard and quietly killing recovery on 2.11 through 3.2.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3765276193


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -158,6 +203,16 @@ def __init__(
         ),
         **kwargs,
     ):
+        if resume_glue_job_on_retry is not None:
+            # Kept as a real named parameter (not **kwargs) so `default_args` still applies correctly.

Review Comment:
   Thanks a lot for this comment! It turned out to affect every operator in the family, not just Glue needed the fix. Added `durable: bool | None = None` as a real named parameter on each glue in this PR & raising a follow up for the rest



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3765766767


##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -139,6 +139,86 @@ To submit a new AWS Glue job you can use :class:`~airflow.providers.amazon.aws.o
   The same AWS IAM role used for the crawler can be used here as well, but it will need
   policies to provide access to the output location for result data.
 
+Durable execution
+==================
+
+``GlueJobOperator`` submits a job run and then polls it to completion on the worker. By default
+the operator runs in a *durable* mode that makes this crash-safe: the Glue job run id is
+persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>` before
+polling begins, so if the worker crashes or is preempted and the task is retried, the operator
+reconnects to the run that is already executing in Glue instead of starting a new one.
+
+This matters more for Glue because a Glue job's ``concurrent_run_limit`` defaults to ``1``, so
+submitting a second run while the first is still active does not create a harmless duplicate, it
+fails outright with ``ConcurrentRunsExceededException`` and the task keeps retrying against a run
+it can never see. Durable execution turns that retry into a normal reconnect.
+
+On retry the operator checks the prior run's state:
+
+* if it is still starting, running, waiting for capacity, or being stopped, the operator
+  reconnects and continues polling
+* if it already succeeded, the operator returns immediately without resubmitting
+* if it failed terminally, or its id has expired and is no longer found, the operator submits the
+  job fresh
+* if it already stopped, the operator submits the job fresh
+
+A stopped run is treated as a failure, not a success. Glue's API has no way to tell a run
+cancelled manually (for example, in the AWS console) apart from one this operator's own
+:meth:`~airflow.providers.amazon.aws.operators.glue.GlueJobOperator.on_kill` stopped, which happens
+whenever ``stop_job_run_on_kill=True`` and the task is killed -- on SIGTERM, on
+``execution_timeout``, or when the task is cleared while running. If the stored state is already
+``STOPPED``, the operator submits fresh. If a reconnect finds the run still stopping and it
+settles into ``STOPPED`` while polling, the operator raises instead of returning a result -- the
+task fails and a normal retry resubmits. Either way, a self-inflicted stop never gets silently
+reported as a false success.
+
+This protection also applies when ``wait_for_completion=False`` -- even though that task attempt
+never polls at all, a retry after a successful submission still reconnects rather than
+resubmitting, since the run id is persisted immediately after submission regardless of whether the
+task waits for it to finish.
+
+Durable execution requires Airflow 3.3 or newer for the task state store lookup above. On earlier
+Airflow versions, or if a prior run was never recorded to the task state store, ``durable=True``
+still recovers a prior run via an older mechanism: the operator checks XCom for a cached run id
+first, then falls back to scanning the job's run history for a run tagged with this task
+instance's identity, and reconnects if it finds one that is still active.
+
+Like the persisted state itself, the stored run id isn't deleted automatically, that only happens

Review Comment:
   Sure, it now mentions both `airflow state-store clean` and `airflow db clean` explicitly



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #71211:
URL: https://github.com/apache/airflow/pull/71211#issuecomment-5200499693

   cc @Alexhans FYI.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3756425116


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +366,150 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable and not self.deferrable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        # The --airflow_task_uuid tag is still written on every nondeferred attempt,
+        # including the first: a later retry's scan can only find it if it was there from the
+        # start. The lookup itself is skipped on the first attempt, since nothing tagged with this
+        # task's UUID could possibly exist yet -- paying for a full run-history scan there would
+        # be pure waste.
+        if self.durable and not self.deferrable and context["ti"].try_number > 1:
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])
+            if existing_job_run_id:
+                self._set_job_run_id(context, existing_job_run_id)
+                return existing_job_run_id
+        self.log.info(
+            "Initializing AWS Glue Job: %s. Wait for completion: %s",
+            self.job_name,
+            self.wait_for_completion,
+        )
+        glue_job_run = self.hook.initialize_job(script_args, self.run_job_kwargs)
+        # Set immediately (before any polling) so on_kill can stop the run even if the worker dies
+        # before poll_until_complete runs.
+        self._set_job_run_id(context, glue_job_run["JobRunId"])
+        # Pushed unconditionally, matching pre-durable-execution behavior: downstream tasks read
+        # this key directly, and it's also the only writer that makes _find_previous_job_run's
+        # XCom tier reachable on the next retry.

Review Comment:
   Reworded — the docstring now says plainly that the XCom tier in 2.x, and that Airflow 3.x always misses there so the scan runs every time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3747219617


##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -139,6 +139,70 @@ To submit a new AWS Glue job you can use :class:`~airflow.providers.amazon.aws.o
   The same AWS IAM role used for the crawler can be used here as well, but it will need
   policies to provide access to the output location for result data.
 
+Durable execution
+==================
+
+``GlueJobOperator`` submits a job run and then polls it to completion on the worker. By default
+the operator runs in a *durable* mode that makes this crash-safe: the Glue job run id is
+persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>` before
+polling begins, so if the worker crashes or is preempted and the task is retried, the operator
+reconnects to the run that is already executing in Glue instead of starting a new one.
+
+This matters more for Glue because a Glue job's ``concurrent_run_limit`` defaults to ``1``, so
+submitting a second run while the first is still active does not create a harmless duplicate, it
+fails outright with ``ConcurrentRunsExceededException`` and the task keeps retrying against a run
+it can never see. Durable execution turns that retry into a normal reconnect.
+
+On retry the operator checks the prior run's state:
+
+* if it is still starting, running, waiting for capacity, or being stopped, the operator
+  reconnects and continues polling
+* if it already succeeded, or was stopped outside Airflow, the operator returns immediately
+  without resubmitting
+* if it failed terminally, or its id has expired and is no longer found, the operator submits the
+  job fresh
+
+A run that was stopped outside Airflow (for example, cancelled manually in the AWS console) is
+treated as a success rather than resubmitted, since the work is genuinely finished, just not the
+way the task expected - the operator logs a warning when this happens.
+
+This protection also applies when ``wait_for_completion=False`` -- even though that task attempt
+never polls at all, a retry after a successful submission still reconnects rather than
+resubmitting, since the run id is persisted immediately after submission regardless of whether the
+task waits for it to finish.
+
+Durable execution requires Airflow 3.3 or newer for the task state store lookup above. On earlier
+Airflow versions, or if the task state store is unavailable at runtime, ``durable=True`` still
+recovers a prior run, just via an older mechanism: the operator checks XCom for a cached run id
+first, then falls back to scanning the job's run history for a run tagged with this task
+instance's identity, and reconnects if it finds one that is still active.
+
+Like the persisted state itself, the stored run id isn't deleted automatically, that only happens
+when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is longer than
+``[state_store] default_retention_days`` (30 days by default) and cleanup runs in between, the run
+id won't be there for the next retry, and the operator falls back to the XCom/scan mechanism
+above rather than reconnecting via task state store. Avoid running cleanup on a schedule shorter
+than your longest ``retry_delay``.
+
+To opt out and always start a fresh run on retry, set ``durable=False``:
+
+.. code-block:: python
+
+  glue_job = GlueJobOperator(
+      task_id="glue_job",
+      job_name="my_glue_job",
+      script_location="s3://glue-examples/glue-scripts/sample_aws_glue_job.py",
+      durable=False,
+  )
+
+Durable execution applies to the synchronous path. When ``deferrable=True`` is set, the Triggerer
+already tracks the run across the wait, so deferrable mode takes precedence and ``durable`` has no

Review Comment:
   Handled in 85398bfb29



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +363,146 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        if self.durable:
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])
+            if existing_job_run_id:
+                self._set_job_run_id(context, existing_job_run_id)
+                return existing_job_run_id
+        self.log.info(
+            "Initializing AWS Glue Job: %s. Wait for completion: %s",
+            self.job_name,
+            self.wait_for_completion,
+        )
+        glue_job_run = self.hook.initialize_job(script_args, self.run_job_kwargs)
+        # Set immediately (before any polling) so on_kill can stop the run even if the worker dies
+        # before poll_until_complete runs.
+        self._set_job_run_id(context, glue_job_run["JobRunId"])
+        return glue_job_run["JobRunId"]
+
+    def get_job_status(self, external_id: JsonValue, context: Context) -> str:
+        """Query the raw job run state; a run id Glue no longer knows about degrades to NOT_FOUND."""
+        job_run_id = cast("str", external_id)
+        # This is the first place a reconnecting attempt learns the run id.
+        self._set_job_run_id(context, job_run_id)
+        try:
+            return self.hook.get_job_state(self.job_name, job_run_id)
+        except ClientError as e:
+            if e.response["Error"]["Code"] == "EntityNotFoundException":
+                return "NOT_FOUND"

Review Comment:
   Handled in 85398bfb29



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3747052566


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +363,146 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        if self.durable:
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])
+            if existing_job_run_id:
+                self._set_job_run_id(context, existing_job_run_id)
+                return existing_job_run_id
+        self.log.info(
+            "Initializing AWS Glue Job: %s. Wait for completion: %s",
+            self.job_name,
+            self.wait_for_completion,
+        )
+        glue_job_run = self.hook.initialize_job(script_args, self.run_job_kwargs)
+        # Set immediately (before any polling) so on_kill can stop the run even if the worker dies
+        # before poll_until_complete runs.
+        self._set_job_run_id(context, glue_job_run["JobRunId"])
+        return glue_job_run["JobRunId"]
+
+    def get_job_status(self, external_id: JsonValue, context: Context) -> str:
+        """Query the raw job run state; a run id Glue no longer knows about degrades to NOT_FOUND."""
+        job_run_id = cast("str", external_id)
+        # This is the first place a reconnecting attempt learns the run id.
+        self._set_job_run_id(context, job_run_id)
+        try:
+            return self.hook.get_job_state(self.job_name, job_run_id)
+        except ClientError as e:
+            if e.response["Error"]["Code"] == "EntityNotFoundException":
+                return "NOT_FOUND"
+            raise
+
+    def is_job_active(self, status: str) -> bool:
+        return status not in (*JOB_RUN_TERMINAL_STATES, NOT_FOUND_STATE)
+
+    def is_job_succeeded(self, status: str) -> bool:
+        if status == "STOPPED":

Review Comment:
   Correct. The fix isnt fixable at the `on_kill` layer the way suggested. `on_kill(self)` misses a context, so it structurally cannot reach `task_state_store` to invalidate anything, and Glue's API gives no way to tell a console cancelled run from one that stopped itself. Since the two cases cannot be distinguished, removed the distinction: `STOPPED` no longer counts as success at all, its now bucketed with `FAILED/TIMEOUT` and always resubmits. 
   
   Trades a rare wasted resubmission (genuine external cancel) for eliminating a silent false success (self-inflicted stop) which is a safer default for correctness.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3765733861


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +364,147 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """Record the run id and surface its console link; guarded so one attempt logs it once."""
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first, then falls back to a task-UUID scan. The XCom tier
+        only works on Airflow 2.11-3.2; Airflow 3.3+ clears task XComs before every non-deferral
+        attempt, so it always misses there and the scan runs every time.
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def _has_stored_external_id(self, context: Context) -> bool:
+        task_state_store = context.get("task_state_store")
+        return task_state_store is not None and task_state_store.get(self.external_id_key) is not None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        # Scan only when there's nothing else to go on: first attempt, no store, or a store that
+        # never recorded this key. A stored id (even terminal) means the caller already decided.
+        if self.durable and context["ti"].try_number > 1 and not self._has_stored_external_id(context):
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])
+            if existing_job_run_id:
+                self._set_job_run_id(context, existing_job_run_id)
+                return existing_job_run_id
+        self.log.info(
+            "Initializing AWS Glue Job: %s. Wait for completion: %s",
+            self.job_name,
+            self.wait_for_completion,
+        )
+        # A prior get_job_status call may have set this to a stale, terminal run id while checking
+        # whether to reconnect. Clear it so on_kill has nothing to act on if initialize_job raises.
+        self._job_run_id = None
+        glue_job_run = self.hook.initialize_job(script_args, self.run_job_kwargs)
+        # Set before polling so on_kill can stop the run even if the worker dies immediately after.
+        self._set_job_run_id(context, glue_job_run["JobRunId"])
+        # Downstream tasks read this key directly; it's also what feeds the XCom tier above.
+        context["ti"].xcom_push(key="glue_job_run_id", value=glue_job_run["JobRunId"])
+        return glue_job_run["JobRunId"]
+
+    def get_job_status(self, external_id: JsonValue, context: Context) -> str:
+        """Query the raw job run state; a run id Glue no longer knows about degrades to NOT_FOUND."""
+        job_run_id = cast("str", external_id)
+        # This is the first place a reconnecting attempt learns the run id.
+        self._set_job_run_id(context, job_run_id)
+        try:
+            return self.hook.get_job_state(self.job_name, job_run_id)
+        except ClientError as e:
+            if e.response["Error"]["Code"] == "EntityNotFoundException":
+                return NOT_FOUND_STATE
+            raise
+
+    def is_job_active(self, status: str) -> bool:
+        return status not in (*JOB_RUN_TERMINAL_STATES, NOT_FOUND_STATE)
+
+    def is_job_succeeded(self, status: str) -> bool:
+        return status in JOB_RUN_SUCCESS_STATES
+
+    def poll_until_complete(self, external_id: JsonValue, context: Context) -> None:
+        job_run_id = cast("str", external_id)
+        self._set_job_run_id(context, job_run_id)
+        if not self.wait_for_completion:
+            self.log.info("AWS Glue Job: %s. Run Id: %s", self.job_name, job_run_id)
+            return
+        glue_job_run = self.hook.job_completion(
+            self.job_name, job_run_id, self.verbose, self.sleep_before_return
+        )
+        state = glue_job_run["JobRunState"]
+        self.log.info("AWS Glue Job: %s status: %s. Run Id: %s", self.job_name, state, job_run_id)
+        if state not in JOB_RUN_SUCCESS_STATES:

Review Comment:
   Moved the STOPPED as failure statement out of the "Durable execution" section entirely and stated plainly that `durable=False` does not change it, since the raise fires unconditionally. 
   
   While tracing this I found the deferrable verbose path (a separate hand-rolled loop in the trigger) still treats STOPPED as success, disagreeing with both the waiter and our fix - filed #71490 for that, out of scope here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3756729227


##########
providers/amazon/tests/unit/amazon/aws/operators/test_glue.py:
##########
@@ -781,6 +784,295 @@ def test_inject_parent_job_info_with_resume_on_retry(
         assert GlueJobOperator.TASK_UUID_ARG in call_args
 
 
+class TestGlueJobOperatorDeprecation:
+    @pytest.mark.parametrize("resume_value", [True, False])
+    def test_warns_on_3_3_plus_and_maps_to_durable(self, resume_value):
+        with mock.patch("airflow.providers.amazon.aws.operators.glue.AIRFLOW_V_3_3_PLUS", True):
+            with pytest.warns(AirflowProviderDeprecationWarning, match="resume_glue_job_on_retry"):
+                glue = GlueJobOperator(
+                    task_id=TASK_ID, job_name=JOB_NAME, resume_glue_job_on_retry=resume_value
+                )
+        assert glue.durable is resume_value
+
+    def test_silent_below_3_3(self):
+        with mock.patch("airflow.providers.amazon.aws.operators.glue.AIRFLOW_V_3_3_PLUS", False):
+            with warnings.catch_warnings(record=True) as caught:
+                warnings.simplefilter("always")
+                glue = GlueJobOperator(task_id=TASK_ID, job_name=JOB_NAME, resume_glue_job_on_retry=True)
+            deprecation_warnings = [
+                w for w in caught if issubclass(w.category, AirflowProviderDeprecationWarning)
+            ]
+        assert deprecation_warnings == []
+        assert glue.durable is True
+
+    def test_both_flags_passed_durable_wins(self):
+        with mock.patch("airflow.providers.amazon.aws.operators.glue.AIRFLOW_V_3_3_PLUS", True):
+            with pytest.warns(AirflowProviderDeprecationWarning):
+                glue = GlueJobOperator(
+                    task_id=TASK_ID,
+                    job_name=JOB_NAME,
+                    durable=True,
+                    resume_glue_job_on_retry=False,
+                )
+        assert glue.durable is True
+
+    @pytest.mark.skipif(
+        not AIRFLOW_V_3_3_PLUS,
+        reason="The <3.3 compat stub's __init__ isn't decorated with BaseOperatorMeta._apply_defaults, "
+        "so default_args injection for durable only works on the real ResumableJobMixin.",
+    )
+    def test_default_args_durable_reaches_operator(self):
+        with DAG(
+            dag_id="test_glue_durable_default_args",
+            schedule=None,
+            start_date=datetime(2024, 1, 1),
+            default_args={"durable": False},
+        ):
+            glue = GlueJobOperator(task_id=TASK_ID, job_name=JOB_NAME)
+        assert glue.durable is False
+
+
+class FakeTaskStateStore:
+    """In-memory task state store for tests."""
+
+    def __init__(self, stored: dict[str, str] | None = None):
+        self._store: dict[str, str] = dict(stored or {})
+
+    def get(self, key: str) -> str | None:
+        return self._store.get(key)
+
+    def set(self, key: str, value: str) -> None:
+        self._store[key] = value
+
+
+@pytest.mark.skipif(
+    not AIRFLOW_V_3_3_PLUS,
+    reason="ResumableJobMixin reconnect requires task_state_store, available in Airflow 3.3+",
+)
+class TestGlueJobOperatorDurableExecution:
+    def _build(self, **kwargs):
+        return GlueJobOperator(task_id=TASK_ID, job_name=JOB_NAME, **kwargs)
+
+    def _stub_empty_scan(self, glue):
+        # submit_job scans for a task UUID tagged run whenever durable is set, regardless of why it
+        # was called. Stub it to return no matches so tests don't depend on that fallback mechanism.
+        glue.hook.conn = mock.MagicMock()
+        glue.hook.conn.get_job_runs.return_value = {"JobRuns": []}
+
+    def _context(self, store=None, try_number=2):
+        ti = mock.MagicMock()
+        ti.try_number = try_number
+        ti.xcom_pull.return_value = None
+        ctx = {"ti": ti}
+        if store is not None:
+            ctx["task_state_store"] = store
+        return ctx
+
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_fresh_submit_persists_before_polling(
+        self, mock_get_conn, mock_initialize_job, mock_job_completion
+    ):
+        glue = self._build(durable=True)
+        self._stub_empty_scan(glue)
+        mock_initialize_job.return_value = {"JobRunId": "jr_new"}
+        store = FakeTaskStateStore()
+        persisted_before_poll = []
+        mock_job_completion.side_effect = lambda *a, **k: (
+            persisted_before_poll.append(store.get("glue_job_run_id")) or {"JobRunState": "SUCCEEDED"}
+        )
+
+        job_run_id = glue.execute(self._context(store))
+
+        assert job_run_id == "jr_new"
+        assert store.get("glue_job_run_id") == "jr_new"
+        assert persisted_before_poll == ["jr_new"]
+        mock_initialize_job.assert_called_once()
+
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "get_job_state")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_reconnect_when_stored_run_is_running(
+        self, mock_get_conn, mock_initialize_job, mock_get_job_state, mock_job_completion
+    ):
+        glue = self._build(durable=True)
+        mock_get_job_state.return_value = "RUNNING"
+        mock_job_completion.return_value = {"JobRunState": "SUCCEEDED"}
+        store = FakeTaskStateStore({"glue_job_run_id": "jr_old"})
+
+        job_run_id = glue.execute(self._context(store))
+
+        assert job_run_id == "jr_old"
+        mock_initialize_job.assert_not_called()
+        mock_job_completion.assert_called_once_with(JOB_NAME, "jr_old", False, 0)
+
+    @pytest.mark.parametrize("status", ["STARTING", "RUNNING", "WAITING", "STOPPING"])
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "get_job_state")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_reconnect_from_every_active_state(
+        self, mock_get_conn, mock_initialize_job, mock_get_job_state, mock_job_completion, status
+    ):
+        glue = self._build(durable=True)
+        mock_get_job_state.return_value = status
+        mock_job_completion.return_value = {"JobRunState": "SUCCEEDED"}
+        store = FakeTaskStateStore({"glue_job_run_id": "jr_old"})
+
+        glue.execute(self._context(store))
+
+        mock_initialize_job.assert_not_called()
+
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "get_job_state")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_already_succeeded_returns_without_resubmit(
+        self, mock_get_conn, mock_initialize_job, mock_get_job_state, mock_job_completion
+    ):
+        glue = self._build(durable=True)
+        mock_get_job_state.return_value = "SUCCEEDED"
+        store = FakeTaskStateStore({"glue_job_run_id": "jr_old"})
+
+        job_run_id = glue.execute(self._context(store))
+
+        assert job_run_id == "jr_old"
+        mock_initialize_job.assert_not_called()
+        mock_job_completion.assert_not_called()
+
+    @pytest.mark.parametrize("status", ["FAILED", "TIMEOUT", "STOPPED"])
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "get_job_state")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_terminal_failure_resubmits_fresh(
+        self, mock_get_conn, mock_initialize_job, mock_get_job_state, mock_job_completion, status
+    ):
+        """STOPPED resubmits like any other terminal state: Glue's API can't tell a console
+        cancellation from a run this operator's own on_kill stopped, so treating it as success
+        would risk silently reporting a self-inflicted stop as done."""
+        glue = self._build(durable=True)
+        self._stub_empty_scan(glue)
+        mock_get_job_state.return_value = status
+        mock_initialize_job.return_value = {"JobRunId": "jr_new"}
+        mock_job_completion.return_value = {"JobRunState": "SUCCEEDED"}
+        store = FakeTaskStateStore({"glue_job_run_id": "jr_old"})
+
+        job_run_id = glue.execute(self._context(store))
+
+        assert job_run_id == "jr_new"
+        assert store.get("glue_job_run_id") == "jr_new"
+        mock_initialize_job.assert_called_once()
+
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "get_job_state")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_not_found_resubmits_fresh(
+        self, mock_get_conn, mock_initialize_job, mock_get_job_state, mock_job_completion
+    ):
+        glue = self._build(durable=True)
+        self._stub_empty_scan(glue)
+        mock_get_job_state.side_effect = ClientError(
+            {"Error": {"Code": "EntityNotFoundException", "Message": "gone"}}, "GetJobRun"
+        )
+        mock_initialize_job.return_value = {"JobRunId": "jr_new"}
+        mock_job_completion.return_value = {"JobRunState": "SUCCEEDED"}
+        store = FakeTaskStateStore({"glue_job_run_id": "jr_old"})
+
+        job_run_id = glue.execute(self._context(store))
+
+        assert job_run_id == "jr_new"
+        mock_initialize_job.assert_called_once()
+
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_durable_false_never_touches_store(self, mock_get_conn, mock_initialize_job, mock_job_completion):
+        glue = self._build(durable=False)
+        mock_initialize_job.return_value = {"JobRunId": "jr_new"}
+        mock_job_completion.return_value = {"JobRunState": "SUCCEEDED"}
+        store = FakeTaskStateStore({"glue_job_run_id": "jr_old"})
+
+        job_run_id = glue.execute(self._context(store))
+
+        assert job_run_id == "jr_new"
+        assert store.get("glue_job_run_id") == "jr_old", "store must be left untouched"
+        mock_initialize_job.assert_called_once()
+
+    @mock.patch.object(GlueJobHook, "conn", new_callable=mock.PropertyMock)
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_first_attempt_skips_the_retry_lookup_entirely(
+        self, mock_get_conn, mock_initialize_job, mock_job_completion, mock_conn
+    ):
+        glue = self._build(durable=True)
+        mock_initialize_job.return_value = {"JobRunId": "jr_new"}
+        mock_job_completion.return_value = {"JobRunState": "SUCCEEDED"}
+        store = FakeTaskStateStore()
+
+        job_run_id = glue.execute(self._context(store, try_number=1))
+
+        assert job_run_id == "jr_new"
+        mock_initialize_job.assert_called_once()
+        mock_conn.return_value.get_job_run.assert_not_called()
+        mock_conn.return_value.get_job_runs.assert_not_called()

Review Comment:
   Added.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3756721953


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +366,150 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable and not self.deferrable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        # The --airflow_task_uuid tag is still written on every nondeferred attempt,
+        # including the first: a later retry's scan can only find it if it was there from the
+        # start. The lookup itself is skipped on the first attempt, since nothing tagged with this
+        # task's UUID could possibly exist yet -- paying for a full run-history scan there would
+        # be pure waste.
+        if self.durable and not self.deferrable and context["ti"].try_number > 1:
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])
+            if existing_job_run_id:
+                self._set_job_run_id(context, existing_job_run_id)
+                return existing_job_run_id
+        self.log.info(
+            "Initializing AWS Glue Job: %s. Wait for completion: %s",
+            self.job_name,
+            self.wait_for_completion,
+        )
+        glue_job_run = self.hook.initialize_job(script_args, self.run_job_kwargs)
+        # Set immediately (before any polling) so on_kill can stop the run even if the worker dies
+        # before poll_until_complete runs.
+        self._set_job_run_id(context, glue_job_run["JobRunId"])
+        # Pushed unconditionally, matching pre-durable-execution behavior: downstream tasks read
+        # this key directly, and it's also the only writer that makes _find_previous_job_run's
+        # XCom tier reachable on the next retry.
+        context["ti"].xcom_push(key="glue_job_run_id", value=glue_job_run["JobRunId"])
+        return glue_job_run["JobRunId"]
+
+    def get_job_status(self, external_id: JsonValue, context: Context) -> str:
+        """Query the raw job run state; a run id Glue no longer knows about degrades to NOT_FOUND."""
+        job_run_id = cast("str", external_id)
+        # This is the first place a reconnecting attempt learns the run id.
+        self._set_job_run_id(context, job_run_id)

Review Comment:
   Good catch. `submit_job` now clears `_job_run_id` right before calling initialise, and `on_kill` is a no op if it's empty. So a failure there leaves nothing stale for on_kill to act on.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3756743898


##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -139,6 +139,79 @@ To submit a new AWS Glue job you can use :class:`~airflow.providers.amazon.aws.o
   The same AWS IAM role used for the crawler can be used here as well, but it will need
   policies to provide access to the output location for result data.
 
+Durable execution
+==================
+
+``GlueJobOperator`` submits a job run and then polls it to completion on the worker. By default
+the operator runs in a *durable* mode that makes this crash-safe: the Glue job run id is
+persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>` before
+polling begins, so if the worker crashes or is preempted and the task is retried, the operator
+reconnects to the run that is already executing in Glue instead of starting a new one.
+
+This matters more for Glue because a Glue job's ``concurrent_run_limit`` defaults to ``1``, so
+submitting a second run while the first is still active does not create a harmless duplicate, it
+fails outright with ``ConcurrentRunsExceededException`` and the task keeps retrying against a run
+it can never see. Durable execution turns that retry into a normal reconnect.
+
+On retry the operator checks the prior run's state:
+
+* if it is still starting, running, waiting for capacity, or being stopped, the operator

Review Comment:
   Handled in 118dacfd80



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +366,150 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable and not self.deferrable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        # The --airflow_task_uuid tag is still written on every nondeferred attempt,
+        # including the first: a later retry's scan can only find it if it was there from the
+        # start. The lookup itself is skipped on the first attempt, since nothing tagged with this
+        # task's UUID could possibly exist yet -- paying for a full run-history scan there would
+        # be pure waste.
+        if self.durable and not self.deferrable and context["ti"].try_number > 1:

Review Comment:
   Handled in 118dacfd80



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +366,150 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable and not self.deferrable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        # The --airflow_task_uuid tag is still written on every nondeferred attempt,
+        # including the first: a later retry's scan can only find it if it was there from the
+        # start. The lookup itself is skipped on the first attempt, since nothing tagged with this
+        # task's UUID could possibly exist yet -- paying for a full run-history scan there would
+        # be pure waste.
+        if self.durable and not self.deferrable and context["ti"].try_number > 1:
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])
+            if existing_job_run_id:
+                self._set_job_run_id(context, existing_job_run_id)
+                return existing_job_run_id
+        self.log.info(
+            "Initializing AWS Glue Job: %s. Wait for completion: %s",
+            self.job_name,
+            self.wait_for_completion,
+        )
+        glue_job_run = self.hook.initialize_job(script_args, self.run_job_kwargs)
+        # Set immediately (before any polling) so on_kill can stop the run even if the worker dies
+        # before poll_until_complete runs.
+        self._set_job_run_id(context, glue_job_run["JobRunId"])
+        # Pushed unconditionally, matching pre-durable-execution behavior: downstream tasks read
+        # this key directly, and it's also the only writer that makes _find_previous_job_run's
+        # XCom tier reachable on the next retry.

Review Comment:
   Handled in 118dacfd80



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3756690354


##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -139,6 +139,79 @@ To submit a new AWS Glue job you can use :class:`~airflow.providers.amazon.aws.o
   The same AWS IAM role used for the crawler can be used here as well, but it will need
   policies to provide access to the output location for result data.
 
+Durable execution
+==================
+
+``GlueJobOperator`` submits a job run and then polls it to completion on the worker. By default
+the operator runs in a *durable* mode that makes this crash-safe: the Glue job run id is
+persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>` before
+polling begins, so if the worker crashes or is preempted and the task is retried, the operator
+reconnects to the run that is already executing in Glue instead of starting a new one.
+
+This matters more for Glue because a Glue job's ``concurrent_run_limit`` defaults to ``1``, so
+submitting a second run while the first is still active does not create a harmless duplicate, it
+fails outright with ``ConcurrentRunsExceededException`` and the task keeps retrying against a run
+it can never see. Durable execution turns that retry into a normal reconnect.
+
+On retry the operator checks the prior run's state:
+
+* if it is still starting, running, waiting for capacity, or being stopped, the operator
+  reconnects and continues polling
+* if it already succeeded, the operator returns immediately without resubmitting
+* if it stopped, failed terminally, or its id has expired and is no longer found, the operator
+  submits the job fresh
+
+A stopped run resubmits rather than being treated as a success. Glue's API has no way to tell a
+run that was cancelled manually (for example, in the AWS console) apart from one this operator's
+own :meth:`~airflow.providers.amazon.aws.operators.glue.GlueJobOperator.on_kill` stopped, which
+happens whenever ``stop_job_run_on_kill=True`` and the task is killed -- on SIGTERM, on
+``execution_timeout``, or when the task is cleared while running. Since the two cases can't be
+told apart, a stopped run always resubmits, so a self-inflicted stop never gets silently reported
+as a false success.
+
+This protection also applies when ``wait_for_completion=False`` -- even though that task attempt
+never polls at all, a retry after a successful submission still reconnects rather than
+resubmitting, since the run id is persisted immediately after submission regardless of whether the
+task waits for it to finish.
+
+Durable execution requires Airflow 3.3 or newer for the task state store lookup above. On earlier
+Airflow versions, or if the task state store is unavailable at runtime, ``durable=True`` still

Review Comment:
   Reworded to name the two cases that actually fall through (no store, or nothing recorded yet), rather than "unavailable" implying it covers the store-errors case too, which it doesn't.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #71211:
URL: https://github.com/apache/airflow/pull/71211#issuecomment-5265846162

   Last round of review handled in 20b6b7550f


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3736165132


##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -139,6 +139,70 @@ To submit a new AWS Glue job you can use :class:`~airflow.providers.amazon.aws.o
   The same AWS IAM role used for the crawler can be used here as well, but it will need
   policies to provide access to the output location for result data.
 
+Durable execution
+==================
+
+``GlueJobOperator`` submits a job run and then polls it to completion on the worker. By default
+the operator runs in a *durable* mode that makes this crash-safe: the Glue job run id is
+persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>` before
+polling begins, so if the worker crashes or is preempted and the task is retried, the operator
+reconnects to the run that is already executing in Glue instead of starting a new one.
+
+This matters more for Glue because a Glue job's ``concurrent_run_limit`` defaults to ``1``, so
+submitting a second run while the first is still active does not create a harmless duplicate, it
+fails outright with ``ConcurrentRunsExceededException`` and the task keeps retrying against a run
+it can never see. Durable execution turns that retry into a normal reconnect.
+
+On retry the operator checks the prior run's state:
+
+* if it is still starting, running, waiting for capacity, or being stopped, the operator
+  reconnects and continues polling
+* if it already succeeded, or was stopped outside Airflow, the operator returns immediately
+  without resubmitting

Review Comment:
   This path also fires when someone clears the task to force a re-run, not just on a retry. The store rows are scoped to `(dag_run_id, task_id, map_index)` and clearing a task doesn't delete them (the only cascade is on `dag_run` deletion), and `[state_store] clear_on_success` defaults to `False`, so the run id is still sitting there after a successful attempt. Clear the task in the UI and the next attempt reads `SUCCEEDED` and goes green in a second or two without submitting anything to Glue.
   
   That's a bigger change than the retry behaviour this section describes, given `durable` now defaults to `True` where `resume_glue_job_on_retry` defaulted to `False`. Was it weighed? At minimum it seems worth documenting here, with `clear_on_success = True` as the way to get clear-to-rerun back.



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +363,146 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        if self.durable:
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])
+            if existing_job_run_id:
+                self._set_job_run_id(context, existing_job_run_id)
+                return existing_job_run_id
+        self.log.info(
+            "Initializing AWS Glue Job: %s. Wait for completion: %s",
+            self.job_name,
+            self.wait_for_completion,
+        )
+        glue_job_run = self.hook.initialize_job(script_args, self.run_job_kwargs)
+        # Set immediately (before any polling) so on_kill can stop the run even if the worker dies
+        # before poll_until_complete runs.
+        self._set_job_run_id(context, glue_job_run["JobRunId"])
+        return glue_job_run["JobRunId"]
+
+    def get_job_status(self, external_id: JsonValue, context: Context) -> str:
+        """Query the raw job run state; a run id Glue no longer knows about degrades to NOT_FOUND."""
+        job_run_id = cast("str", external_id)
+        # This is the first place a reconnecting attempt learns the run id.
+        self._set_job_run_id(context, job_run_id)
+        try:
+            return self.hook.get_job_state(self.job_name, job_run_id)
+        except ClientError as e:
+            if e.response["Error"]["Code"] == "EntityNotFoundException":
+                return "NOT_FOUND"
+            raise
+
+    def is_job_active(self, status: str) -> bool:
+        return status not in (*JOB_RUN_TERMINAL_STATES, NOT_FOUND_STATE)
+
+    def is_job_succeeded(self, status: str) -> bool:
+        if status == "STOPPED":

Review Comment:
   `STOPPED` isn't only reachable from a manual console stop. `on_kill` stops the run itself when `stop_job_run_on_kill=True`, and `on_kill` fires both on SIGTERM and on `execution_timeout` (`task_runner.py` calls `task.on_kill()` in both), which leave the task `UP_FOR_RETRY` when retries remain. So a task that trips `execution_timeout` stops its own Glue run, and the retry then reads `STOPPED` and reports success with the ETL half done. Before this PR that retry resubmitted.
   
   The warning also says "stopped outside Airflow", which is the one case it isn't. Could `on_kill` drop the stored run id when it stops the run, so the retry resubmits instead of inheriting a `STOPPED` it caused itself?



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +363,146 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        if self.durable:
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])

Review Comment:
   This runs on the first attempt too, not just retries, and with `durable` defaulting to `True` every Glue task now pays it. Because nothing pushes the XCom key any more, `_find_previous_job_run` always takes the scan branch, and `_find_job_run_id_by_task_uuid` paginates the entire run history at 50 per page when there's no match, so a busy job is dozens of `GetJobRuns` calls before every submission. It also runs on the deferrable path, since `execute` calls `submit_job` there.
   
   Can it be gated on the attempt actually being a retry (`context["ti"].try_number > 1`), or on the task state store being absent, rather than on `durable` alone? Line 174 of the docs describes it as the fallback for when the store is unavailable, which is what I expected the code to do.



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +363,146 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        if self.durable:
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])
+            if existing_job_run_id:
+                self._set_job_run_id(context, existing_job_run_id)
+                return existing_job_run_id
+        self.log.info(
+            "Initializing AWS Glue Job: %s. Wait for completion: %s",
+            self.job_name,
+            self.wait_for_completion,
+        )
+        glue_job_run = self.hook.initialize_job(script_args, self.run_job_kwargs)
+        # Set immediately (before any polling) so on_kill can stop the run even if the worker dies
+        # before poll_until_complete runs.
+        self._set_job_run_id(context, glue_job_run["JobRunId"])

Review Comment:
   The `context["ti"].xcom_push(key="glue_job_run_id", ...)` that main does right after `initialize_job` is gone, and `_set_job_run_id` doesn't replace it (it sets `self._job_run_id` and persists the link; the SDK mixin writes only to `task_state_store`). Two consequences.
   
   Downstream tasks doing `xcom_pull(task_ids="glue_job", key="glue_job_run_id")` stop getting a value. The `return_value` XCom still carries the id, but the explicit key is what people wrote against.
   
   And the XCom tier in `_find_previous_job_run` can never hit now: the push inside the scan branch is the only remaining writer, so the `xcom_pull` above it is always `None` and every durable attempt falls through to the history scan. The description says the XCom-then-scan mechanism is kept exactly as it was, but the write half was dropped. The assertions that guarded the push (`assert len(xcom_calls) == 1, "Should push new glue_job_run_id"`) were deleted from `test_check_previous_job_id_run_new_on_finished` in this PR, which is why nothing caught it.



##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -139,6 +139,70 @@ To submit a new AWS Glue job you can use :class:`~airflow.providers.amazon.aws.o
   The same AWS IAM role used for the crawler can be used here as well, but it will need
   policies to provide access to the output location for result data.
 
+Durable execution
+==================
+
+``GlueJobOperator`` submits a job run and then polls it to completion on the worker. By default
+the operator runs in a *durable* mode that makes this crash-safe: the Glue job run id is
+persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>` before
+polling begins, so if the worker crashes or is preempted and the task is retried, the operator
+reconnects to the run that is already executing in Glue instead of starting a new one.
+
+This matters more for Glue because a Glue job's ``concurrent_run_limit`` defaults to ``1``, so
+submitting a second run while the first is still active does not create a harmless duplicate, it
+fails outright with ``ConcurrentRunsExceededException`` and the task keeps retrying against a run
+it can never see. Durable execution turns that retry into a normal reconnect.
+
+On retry the operator checks the prior run's state:
+
+* if it is still starting, running, waiting for capacity, or being stopped, the operator
+  reconnects and continues polling
+* if it already succeeded, or was stopped outside Airflow, the operator returns immediately
+  without resubmitting
+* if it failed terminally, or its id has expired and is no longer found, the operator submits the
+  job fresh
+
+A run that was stopped outside Airflow (for example, cancelled manually in the AWS console) is
+treated as a success rather than resubmitted, since the work is genuinely finished, just not the
+way the task expected - the operator logs a warning when this happens.
+
+This protection also applies when ``wait_for_completion=False`` -- even though that task attempt
+never polls at all, a retry after a successful submission still reconnects rather than
+resubmitting, since the run id is persisted immediately after submission regardless of whether the
+task waits for it to finish.
+
+Durable execution requires Airflow 3.3 or newer for the task state store lookup above. On earlier
+Airflow versions, or if the task state store is unavailable at runtime, ``durable=True`` still
+recovers a prior run, just via an older mechanism: the operator checks XCom for a cached run id
+first, then falls back to scanning the job's run history for a run tagged with this task
+instance's identity, and reconnects if it finds one that is still active.
+
+Like the persisted state itself, the stored run id isn't deleted automatically, that only happens
+when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is longer than
+``[state_store] default_retention_days`` (30 days by default) and cleanup runs in between, the run
+id won't be there for the next retry, and the operator falls back to the XCom/scan mechanism
+above rather than reconnecting via task state store. Avoid running cleanup on a schedule shorter
+than your longest ``retry_delay``.
+
+To opt out and always start a fresh run on retry, set ``durable=False``:
+
+.. code-block:: python
+
+  glue_job = GlueJobOperator(
+      task_id="glue_job",
+      job_name="my_glue_job",
+      script_location="s3://glue-examples/glue-scripts/sample_aws_glue_job.py",
+      durable=False,
+  )
+
+Durable execution applies to the synchronous path. When ``deferrable=True`` is set, the Triggerer
+already tracks the run across the wait, so deferrable mode takes precedence and ``durable`` has no

Review Comment:
   `durable` isn't fully inert on the deferrable path. `execute` calls `submit_job`, which for `durable=True` (the new default) injects `--airflow_task_uuid` into every run's arguments and runs the reconnect lookup before submitting. Deferrable Glue runs get an extra script arg and a history scan they didn't have before. Either gate `_build_script_args`/`_find_previous_job_run` on `not self.deferrable`, or say here what deferrable plus durable actually does.



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +363,146 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        if self.durable:
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])
+            if existing_job_run_id:
+                self._set_job_run_id(context, existing_job_run_id)
+                return existing_job_run_id
+        self.log.info(
+            "Initializing AWS Glue Job: %s. Wait for completion: %s",
+            self.job_name,
+            self.wait_for_completion,
+        )
+        glue_job_run = self.hook.initialize_job(script_args, self.run_job_kwargs)
+        # Set immediately (before any polling) so on_kill can stop the run even if the worker dies
+        # before poll_until_complete runs.
+        self._set_job_run_id(context, glue_job_run["JobRunId"])
+        return glue_job_run["JobRunId"]
+
+    def get_job_status(self, external_id: JsonValue, context: Context) -> str:
+        """Query the raw job run state; a run id Glue no longer knows about degrades to NOT_FOUND."""
+        job_run_id = cast("str", external_id)
+        # This is the first place a reconnecting attempt learns the run id.
+        self._set_job_run_id(context, job_run_id)
+        try:
+            return self.hook.get_job_state(self.job_name, job_run_id)
+        except ClientError as e:
+            if e.response["Error"]["Code"] == "EntityNotFoundException":
+                return "NOT_FOUND"

Review Comment:
   `NOT_FOUND_STATE` is defined above and used in `is_job_active`, but this returns the literal. Worth using the constant so the two can't drift, since a mismatch makes `is_job_active` treat a deleted run as active.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3756149780


##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -139,6 +139,79 @@ To submit a new AWS Glue job you can use :class:`~airflow.providers.amazon.aws.o
   The same AWS IAM role used for the crawler can be used here as well, but it will need
   policies to provide access to the output location for result data.
 
+Durable execution
+==================
+
+``GlueJobOperator`` submits a job run and then polls it to completion on the worker. By default
+the operator runs in a *durable* mode that makes this crash-safe: the Glue job run id is
+persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>` before
+polling begins, so if the worker crashes or is preempted and the task is retried, the operator
+reconnects to the run that is already executing in Glue instead of starting a new one.
+
+This matters more for Glue because a Glue job's ``concurrent_run_limit`` defaults to ``1``, so
+submitting a second run while the first is still active does not create a harmless duplicate, it
+fails outright with ``ConcurrentRunsExceededException`` and the task keeps retrying against a run
+it can never see. Durable execution turns that retry into a normal reconnect.
+
+On retry the operator checks the prior run's state:
+
+* if it is still starting, running, waiting for capacity, or being stopped, the operator

Review Comment:
   Fixed to match the actual fix from previous comment: STOPPING still reconnects and polls, a directly stored STOPPED resubmits, and a reconnect that settles into STOPPED mid-poll now raises rather than either resubmitting or succeeding.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3756567950


##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -139,6 +139,79 @@ To submit a new AWS Glue job you can use :class:`~airflow.providers.amazon.aws.o
   The same AWS IAM role used for the crawler can be used here as well, but it will need
   policies to provide access to the output location for result data.
 
+Durable execution
+==================
+
+``GlueJobOperator`` submits a job run and then polls it to completion on the worker. By default
+the operator runs in a *durable* mode that makes this crash-safe: the Glue job run id is
+persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>` before
+polling begins, so if the worker crashes or is preempted and the task is retried, the operator
+reconnects to the run that is already executing in Glue instead of starting a new one.
+
+This matters more for Glue because a Glue job's ``concurrent_run_limit`` defaults to ``1``, so
+submitting a second run while the first is still active does not create a harmless duplicate, it
+fails outright with ``ConcurrentRunsExceededException`` and the task keeps retrying against a run
+it can never see. Durable execution turns that retry into a normal reconnect.
+
+On retry the operator checks the prior run's state:
+
+* if it is still starting, running, waiting for capacity, or being stopped, the operator
+  reconnects and continues polling
+* if it already succeeded, the operator returns immediately without resubmitting
+* if it stopped, failed terminally, or its id has expired and is no longer found, the operator
+  submits the job fresh
+
+A stopped run resubmits rather than being treated as a success. Glue's API has no way to tell a
+run that was cancelled manually (for example, in the AWS console) apart from one this operator's
+own :meth:`~airflow.providers.amazon.aws.operators.glue.GlueJobOperator.on_kill` stopped, which
+happens whenever ``stop_job_run_on_kill=True`` and the task is killed -- on SIGTERM, on
+``execution_timeout``, or when the task is cleared while running. Since the two cases can't be
+told apart, a stopped run always resubmits, so a self-inflicted stop never gets silently reported
+as a false success.
+
+This protection also applies when ``wait_for_completion=False`` -- even though that task attempt
+never polls at all, a retry after a successful submission still reconnects rather than
+resubmitting, since the run id is persisted immediately after submission regardless of whether the
+task waits for it to finish.
+
+Durable execution requires Airflow 3.3 or newer for the task state store lookup above. On earlier
+Airflow versions, or if the task state store is unavailable at runtime, ``durable=True`` still
+recovers a prior run, just via an older mechanism: the operator checks XCom for a cached run id
+first, then falls back to scanning the job's run history for a run tagged with this task
+instance's identity, and reconnects if it finds one that is still active.
+
+Like the persisted state itself, the stored run id isn't deleted automatically, that only happens
+when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is longer than
+``[state_store] default_retention_days`` (30 days by default) and cleanup runs in between, the run
+id won't be there for the next retry, and the operator falls back to the XCom/scan mechanism
+above rather than reconnecting via task state store. Avoid running cleanup on a schedule shorter
+than your longest ``retry_delay``.
+
+Clearing a task is treated the same as a retry, which matters specifically for a task whose job
+already succeeded: clearing does not delete the stored run id, so the next attempt reads it back
+and returns immediately without submitting anything to Glue. See
+:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the

Review Comment:
   Its handled by #71356 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3756745715


##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -139,6 +139,79 @@ To submit a new AWS Glue job you can use :class:`~airflow.providers.amazon.aws.o
   The same AWS IAM role used for the crawler can be used here as well, but it will need
   policies to provide access to the output location for result data.
 
+Durable execution
+==================
+
+``GlueJobOperator`` submits a job run and then polls it to completion on the worker. By default
+the operator runs in a *durable* mode that makes this crash-safe: the Glue job run id is
+persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>` before
+polling begins, so if the worker crashes or is preempted and the task is retried, the operator
+reconnects to the run that is already executing in Glue instead of starting a new one.
+
+This matters more for Glue because a Glue job's ``concurrent_run_limit`` defaults to ``1``, so
+submitting a second run while the first is still active does not create a harmless duplicate, it
+fails outright with ``ConcurrentRunsExceededException`` and the task keeps retrying against a run
+it can never see. Durable execution turns that retry into a normal reconnect.
+
+On retry the operator checks the prior run's state:
+
+* if it is still starting, running, waiting for capacity, or being stopped, the operator
+  reconnects and continues polling
+* if it already succeeded, the operator returns immediately without resubmitting
+* if it stopped, failed terminally, or its id has expired and is no longer found, the operator
+  submits the job fresh
+
+A stopped run resubmits rather than being treated as a success. Glue's API has no way to tell a
+run that was cancelled manually (for example, in the AWS console) apart from one this operator's
+own :meth:`~airflow.providers.amazon.aws.operators.glue.GlueJobOperator.on_kill` stopped, which
+happens whenever ``stop_job_run_on_kill=True`` and the task is killed -- on SIGTERM, on
+``execution_timeout``, or when the task is cleared while running. Since the two cases can't be
+told apart, a stopped run always resubmits, so a self-inflicted stop never gets silently reported
+as a false success.
+
+This protection also applies when ``wait_for_completion=False`` -- even though that task attempt
+never polls at all, a retry after a successful submission still reconnects rather than
+resubmitting, since the run id is persisted immediately after submission regardless of whether the
+task waits for it to finish.
+
+Durable execution requires Airflow 3.3 or newer for the task state store lookup above. On earlier
+Airflow versions, or if the task state store is unavailable at runtime, ``durable=True`` still
+recovers a prior run, just via an older mechanism: the operator checks XCom for a cached run id
+first, then falls back to scanning the job's run history for a run tagged with this task
+instance's identity, and reconnects if it finds one that is still active.
+
+Like the persisted state itself, the stored run id isn't deleted automatically, that only happens
+when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is longer than
+``[state_store] default_retention_days`` (30 days by default) and cleanup runs in between, the run
+id won't be there for the next retry, and the operator falls back to the XCom/scan mechanism
+above rather than reconnecting via task state store. Avoid running cleanup on a schedule shorter
+than your longest ``retry_delay``.
+
+Clearing a task is treated the same as a retry, which matters specifically for a task whose job
+already succeeded: clearing does not delete the stored run id, so the next attempt reads it back
+and returns immediately without submitting anything to Glue. See
+:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
+``[state_store] clear_on_success`` setting that restores "clearing always resubmits."
+
+To opt out and always start a fresh run on retry, set ``durable=False``:
+
+.. code-block:: python
+
+  glue_job = GlueJobOperator(
+      task_id="glue_job",
+      job_name="my_glue_job",
+      script_location="s3://glue-examples/glue-scripts/sample_aws_glue_job.py",
+      durable=False,
+  )
+
+Durable execution applies to the synchronous path. When ``deferrable=True`` is set, the Triggerer
+already tracks the run across the wait, so deferrable mode takes precedence and ``durable`` has no

Review Comment:
   Handled in 118dacfd80



##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -139,6 +139,79 @@ To submit a new AWS Glue job you can use :class:`~airflow.providers.amazon.aws.o
   The same AWS IAM role used for the crawler can be used here as well, but it will need
   policies to provide access to the output location for result data.
 
+Durable execution
+==================
+
+``GlueJobOperator`` submits a job run and then polls it to completion on the worker. By default
+the operator runs in a *durable* mode that makes this crash-safe: the Glue job run id is
+persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>` before
+polling begins, so if the worker crashes or is preempted and the task is retried, the operator
+reconnects to the run that is already executing in Glue instead of starting a new one.
+
+This matters more for Glue because a Glue job's ``concurrent_run_limit`` defaults to ``1``, so
+submitting a second run while the first is still active does not create a harmless duplicate, it
+fails outright with ``ConcurrentRunsExceededException`` and the task keeps retrying against a run
+it can never see. Durable execution turns that retry into a normal reconnect.
+
+On retry the operator checks the prior run's state:
+
+* if it is still starting, running, waiting for capacity, or being stopped, the operator
+  reconnects and continues polling
+* if it already succeeded, the operator returns immediately without resubmitting
+* if it stopped, failed terminally, or its id has expired and is no longer found, the operator
+  submits the job fresh
+
+A stopped run resubmits rather than being treated as a success. Glue's API has no way to tell a
+run that was cancelled manually (for example, in the AWS console) apart from one this operator's
+own :meth:`~airflow.providers.amazon.aws.operators.glue.GlueJobOperator.on_kill` stopped, which
+happens whenever ``stop_job_run_on_kill=True`` and the task is killed -- on SIGTERM, on
+``execution_timeout``, or when the task is cleared while running. Since the two cases can't be
+told apart, a stopped run always resubmits, so a self-inflicted stop never gets silently reported
+as a false success.
+
+This protection also applies when ``wait_for_completion=False`` -- even though that task attempt
+never polls at all, a retry after a successful submission still reconnects rather than
+resubmitting, since the run id is persisted immediately after submission regardless of whether the
+task waits for it to finish.
+
+Durable execution requires Airflow 3.3 or newer for the task state store lookup above. On earlier
+Airflow versions, or if the task state store is unavailable at runtime, ``durable=True`` still

Review Comment:
   Handled in 118dacfd80



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +366,150 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """
+        Record the run id and surface its console link.
+
+        Called from every path that learns a run id (fresh submit, bootstrap search, reconnect) so
+        ``on_kill`` can stop the run and the link is available before polling starts. Guarded on the
+        id changing, so a single attempt logs the link once.
+        """
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable and not self.deferrable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first; falls back to a task-UUID scan of the job's run
+        history when XCom has nothing (e.g. the cached id expired, or this is the first retry to
+        run under a provider version that persists it).
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        # The --airflow_task_uuid tag is still written on every nondeferred attempt,
+        # including the first: a later retry's scan can only find it if it was there from the
+        # start. The lookup itself is skipped on the first attempt, since nothing tagged with this
+        # task's UUID could possibly exist yet -- paying for a full run-history scan there would
+        # be pure waste.
+        if self.durable and not self.deferrable and context["ti"].try_number > 1:
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])
+            if existing_job_run_id:
+                self._set_job_run_id(context, existing_job_run_id)
+                return existing_job_run_id
+        self.log.info(
+            "Initializing AWS Glue Job: %s. Wait for completion: %s",
+            self.job_name,
+            self.wait_for_completion,
+        )
+        glue_job_run = self.hook.initialize_job(script_args, self.run_job_kwargs)
+        # Set immediately (before any polling) so on_kill can stop the run even if the worker dies
+        # before poll_until_complete runs.
+        self._set_job_run_id(context, glue_job_run["JobRunId"])
+        # Pushed unconditionally, matching pre-durable-execution behavior: downstream tasks read
+        # this key directly, and it's also the only writer that makes _find_previous_job_run's
+        # XCom tier reachable on the next retry.
+        context["ti"].xcom_push(key="glue_job_run_id", value=glue_job_run["JobRunId"])
+        return glue_job_run["JobRunId"]
+
+    def get_job_status(self, external_id: JsonValue, context: Context) -> str:
+        """Query the raw job run state; a run id Glue no longer knows about degrades to NOT_FOUND."""
+        job_run_id = cast("str", external_id)
+        # This is the first place a reconnecting attempt learns the run id.
+        self._set_job_run_id(context, job_run_id)

Review Comment:
   Handled in 118dacfd80



##########
providers/amazon/tests/unit/amazon/aws/operators/test_glue.py:
##########
@@ -781,6 +784,295 @@ def test_inject_parent_job_info_with_resume_on_retry(
         assert GlueJobOperator.TASK_UUID_ARG in call_args
 
 
+class TestGlueJobOperatorDeprecation:
+    @pytest.mark.parametrize("resume_value", [True, False])
+    def test_warns_on_3_3_plus_and_maps_to_durable(self, resume_value):
+        with mock.patch("airflow.providers.amazon.aws.operators.glue.AIRFLOW_V_3_3_PLUS", True):
+            with pytest.warns(AirflowProviderDeprecationWarning, match="resume_glue_job_on_retry"):
+                glue = GlueJobOperator(
+                    task_id=TASK_ID, job_name=JOB_NAME, resume_glue_job_on_retry=resume_value
+                )
+        assert glue.durable is resume_value
+
+    def test_silent_below_3_3(self):
+        with mock.patch("airflow.providers.amazon.aws.operators.glue.AIRFLOW_V_3_3_PLUS", False):
+            with warnings.catch_warnings(record=True) as caught:
+                warnings.simplefilter("always")
+                glue = GlueJobOperator(task_id=TASK_ID, job_name=JOB_NAME, resume_glue_job_on_retry=True)
+            deprecation_warnings = [
+                w for w in caught if issubclass(w.category, AirflowProviderDeprecationWarning)
+            ]
+        assert deprecation_warnings == []
+        assert glue.durable is True
+
+    def test_both_flags_passed_durable_wins(self):
+        with mock.patch("airflow.providers.amazon.aws.operators.glue.AIRFLOW_V_3_3_PLUS", True):
+            with pytest.warns(AirflowProviderDeprecationWarning):
+                glue = GlueJobOperator(
+                    task_id=TASK_ID,
+                    job_name=JOB_NAME,
+                    durable=True,
+                    resume_glue_job_on_retry=False,
+                )
+        assert glue.durable is True
+
+    @pytest.mark.skipif(
+        not AIRFLOW_V_3_3_PLUS,
+        reason="The <3.3 compat stub's __init__ isn't decorated with BaseOperatorMeta._apply_defaults, "
+        "so default_args injection for durable only works on the real ResumableJobMixin.",
+    )
+    def test_default_args_durable_reaches_operator(self):
+        with DAG(
+            dag_id="test_glue_durable_default_args",
+            schedule=None,
+            start_date=datetime(2024, 1, 1),
+            default_args={"durable": False},
+        ):
+            glue = GlueJobOperator(task_id=TASK_ID, job_name=JOB_NAME)
+        assert glue.durable is False
+
+
+class FakeTaskStateStore:
+    """In-memory task state store for tests."""
+
+    def __init__(self, stored: dict[str, str] | None = None):
+        self._store: dict[str, str] = dict(stored or {})
+
+    def get(self, key: str) -> str | None:
+        return self._store.get(key)
+
+    def set(self, key: str, value: str) -> None:
+        self._store[key] = value
+
+
+@pytest.mark.skipif(
+    not AIRFLOW_V_3_3_PLUS,
+    reason="ResumableJobMixin reconnect requires task_state_store, available in Airflow 3.3+",
+)
+class TestGlueJobOperatorDurableExecution:
+    def _build(self, **kwargs):
+        return GlueJobOperator(task_id=TASK_ID, job_name=JOB_NAME, **kwargs)
+
+    def _stub_empty_scan(self, glue):
+        # submit_job scans for a task UUID tagged run whenever durable is set, regardless of why it
+        # was called. Stub it to return no matches so tests don't depend on that fallback mechanism.
+        glue.hook.conn = mock.MagicMock()
+        glue.hook.conn.get_job_runs.return_value = {"JobRuns": []}
+
+    def _context(self, store=None, try_number=2):
+        ti = mock.MagicMock()
+        ti.try_number = try_number
+        ti.xcom_pull.return_value = None
+        ctx = {"ti": ti}
+        if store is not None:
+            ctx["task_state_store"] = store
+        return ctx
+
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_fresh_submit_persists_before_polling(
+        self, mock_get_conn, mock_initialize_job, mock_job_completion
+    ):
+        glue = self._build(durable=True)
+        self._stub_empty_scan(glue)
+        mock_initialize_job.return_value = {"JobRunId": "jr_new"}
+        store = FakeTaskStateStore()
+        persisted_before_poll = []
+        mock_job_completion.side_effect = lambda *a, **k: (
+            persisted_before_poll.append(store.get("glue_job_run_id")) or {"JobRunState": "SUCCEEDED"}
+        )
+
+        job_run_id = glue.execute(self._context(store))
+
+        assert job_run_id == "jr_new"
+        assert store.get("glue_job_run_id") == "jr_new"
+        assert persisted_before_poll == ["jr_new"]
+        mock_initialize_job.assert_called_once()
+
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "get_job_state")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_reconnect_when_stored_run_is_running(
+        self, mock_get_conn, mock_initialize_job, mock_get_job_state, mock_job_completion
+    ):
+        glue = self._build(durable=True)
+        mock_get_job_state.return_value = "RUNNING"
+        mock_job_completion.return_value = {"JobRunState": "SUCCEEDED"}
+        store = FakeTaskStateStore({"glue_job_run_id": "jr_old"})
+
+        job_run_id = glue.execute(self._context(store))
+
+        assert job_run_id == "jr_old"
+        mock_initialize_job.assert_not_called()
+        mock_job_completion.assert_called_once_with(JOB_NAME, "jr_old", False, 0)
+
+    @pytest.mark.parametrize("status", ["STARTING", "RUNNING", "WAITING", "STOPPING"])
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "get_job_state")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_reconnect_from_every_active_state(
+        self, mock_get_conn, mock_initialize_job, mock_get_job_state, mock_job_completion, status
+    ):
+        glue = self._build(durable=True)
+        mock_get_job_state.return_value = status
+        mock_job_completion.return_value = {"JobRunState": "SUCCEEDED"}
+        store = FakeTaskStateStore({"glue_job_run_id": "jr_old"})
+
+        glue.execute(self._context(store))
+
+        mock_initialize_job.assert_not_called()
+
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "get_job_state")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_already_succeeded_returns_without_resubmit(
+        self, mock_get_conn, mock_initialize_job, mock_get_job_state, mock_job_completion
+    ):
+        glue = self._build(durable=True)
+        mock_get_job_state.return_value = "SUCCEEDED"
+        store = FakeTaskStateStore({"glue_job_run_id": "jr_old"})
+
+        job_run_id = glue.execute(self._context(store))
+
+        assert job_run_id == "jr_old"
+        mock_initialize_job.assert_not_called()
+        mock_job_completion.assert_not_called()
+
+    @pytest.mark.parametrize("status", ["FAILED", "TIMEOUT", "STOPPED"])
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "get_job_state")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_terminal_failure_resubmits_fresh(
+        self, mock_get_conn, mock_initialize_job, mock_get_job_state, mock_job_completion, status
+    ):
+        """STOPPED resubmits like any other terminal state: Glue's API can't tell a console
+        cancellation from a run this operator's own on_kill stopped, so treating it as success
+        would risk silently reporting a self-inflicted stop as done."""
+        glue = self._build(durable=True)
+        self._stub_empty_scan(glue)
+        mock_get_job_state.return_value = status
+        mock_initialize_job.return_value = {"JobRunId": "jr_new"}
+        mock_job_completion.return_value = {"JobRunState": "SUCCEEDED"}
+        store = FakeTaskStateStore({"glue_job_run_id": "jr_old"})
+
+        job_run_id = glue.execute(self._context(store))
+
+        assert job_run_id == "jr_new"
+        assert store.get("glue_job_run_id") == "jr_new"
+        mock_initialize_job.assert_called_once()
+
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "get_job_state")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_not_found_resubmits_fresh(
+        self, mock_get_conn, mock_initialize_job, mock_get_job_state, mock_job_completion
+    ):
+        glue = self._build(durable=True)
+        self._stub_empty_scan(glue)
+        mock_get_job_state.side_effect = ClientError(
+            {"Error": {"Code": "EntityNotFoundException", "Message": "gone"}}, "GetJobRun"
+        )
+        mock_initialize_job.return_value = {"JobRunId": "jr_new"}
+        mock_job_completion.return_value = {"JobRunState": "SUCCEEDED"}
+        store = FakeTaskStateStore({"glue_job_run_id": "jr_old"})
+
+        job_run_id = glue.execute(self._context(store))
+
+        assert job_run_id == "jr_new"
+        mock_initialize_job.assert_called_once()
+
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_durable_false_never_touches_store(self, mock_get_conn, mock_initialize_job, mock_job_completion):
+        glue = self._build(durable=False)
+        mock_initialize_job.return_value = {"JobRunId": "jr_new"}
+        mock_job_completion.return_value = {"JobRunState": "SUCCEEDED"}
+        store = FakeTaskStateStore({"glue_job_run_id": "jr_old"})
+
+        job_run_id = glue.execute(self._context(store))
+
+        assert job_run_id == "jr_new"
+        assert store.get("glue_job_run_id") == "jr_old", "store must be left untouched"
+        mock_initialize_job.assert_called_once()
+
+    @mock.patch.object(GlueJobHook, "conn", new_callable=mock.PropertyMock)
+    @mock.patch.object(GlueJobHook, "job_completion")
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_first_attempt_skips_the_retry_lookup_entirely(
+        self, mock_get_conn, mock_initialize_job, mock_job_completion, mock_conn
+    ):
+        glue = self._build(durable=True)
+        mock_initialize_job.return_value = {"JobRunId": "jr_new"}
+        mock_job_completion.return_value = {"JobRunState": "SUCCEEDED"}
+        store = FakeTaskStateStore()
+
+        job_run_id = glue.execute(self._context(store, try_number=1))
+
+        assert job_run_id == "jr_new"
+        mock_initialize_job.assert_called_once()
+        mock_conn.return_value.get_job_run.assert_not_called()
+        mock_conn.return_value.get_job_runs.assert_not_called()

Review Comment:
   Handled in 118dacfd80



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3764425760


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +364,147 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """Record the run id and surface its console link; guarded so one attempt logs it once."""
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first, then falls back to a task-UUID scan. The XCom tier
+        only works on Airflow 2.11-3.2; Airflow 3.3+ clears task XComs before every non-deferral
+        attempt, so it always misses there and the scan runs every time.
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def _has_stored_external_id(self, context: Context) -> bool:
+        task_state_store = context.get("task_state_store")
+        return task_state_store is not None and task_state_store.get(self.external_id_key) is not None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        # Scan only when there's nothing else to go on: first attempt, no store, or a store that
+        # never recorded this key. A stored id (even terminal) means the caller already decided.
+        if self.durable and context["ti"].try_number > 1 and not self._has_stored_external_id(context):
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])
+            if existing_job_run_id:
+                self._set_job_run_id(context, existing_job_run_id)
+                return existing_job_run_id
+        self.log.info(
+            "Initializing AWS Glue Job: %s. Wait for completion: %s",
+            self.job_name,
+            self.wait_for_completion,
+        )
+        # A prior get_job_status call may have set this to a stale, terminal run id while checking
+        # whether to reconnect. Clear it so on_kill has nothing to act on if initialize_job raises.
+        self._job_run_id = None
+        glue_job_run = self.hook.initialize_job(script_args, self.run_job_kwargs)
+        # Set before polling so on_kill can stop the run even if the worker dies immediately after.
+        self._set_job_run_id(context, glue_job_run["JobRunId"])
+        # Downstream tasks read this key directly; it's also what feeds the XCom tier above.
+        context["ti"].xcom_push(key="glue_job_run_id", value=glue_job_run["JobRunId"])

Review Comment:
   Follow-up on my round-1 comment. The push is back, but only on the branch that submits. `execute_resumable` returns from the reconnect branch and from the already-succeeded branch (`task-sdk/src/airflow/sdk/bases/resumablejobmixin.py:196-199`) without ever reaching `submit_job`, and every non-deferral attempt clears the task instance's XComs first. So after a retry that reconnects, or a clear that short-circuits on an already-succeeded run, `xcom_pull(task_ids="glue_job", key="glue_job_run_id")` returns None.
   
   The asymmetry is what makes it read as an oversight: `_set_job_run_id` already carries the other per-attempt side effect (`GlueJobRunDetailsLink.persist`) on all four paths, so the link survives and only the run-id key was left behind.
   
   Scope, stated honestly: `execute` still returns the run id, so `glue_job.output` and keyless `xcom_pull` are unaffected, and there is no in-repo consumer of the key. It is a de-facto contract rather than a documented one, but it is the one the comment on the line above asserts ("Downstream tasks read this key directly").
   
   Ask: move the push into `_set_job_run_id` beside the `persist` call and drop it here. It is already called on all four paths, already de-duplicated by the `if self._job_run_id == job_run_id: return` guard, and same-key `xcom_push` is delete-then-insert, so a stale-then-fresh sequence still ends on the correct id, exactly as the link already does.



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +364,147 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """Record the run id and surface its console link; guarded so one attempt logs it once."""
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first, then falls back to a task-UUID scan. The XCom tier
+        only works on Airflow 2.11-3.2; Airflow 3.3+ clears task XComs before every non-deferral
+        attempt, so it always misses there and the scan runs every time.
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def _has_stored_external_id(self, context: Context) -> bool:
+        task_state_store = context.get("task_state_store")
+        return task_state_store is not None and task_state_store.get(self.external_id_key) is not None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        # Scan only when there's nothing else to go on: first attempt, no store, or a store that
+        # never recorded this key. A stored id (even terminal) means the caller already decided.
+        if self.durable and context["ti"].try_number > 1 and not self._has_stored_external_id(context):

Review Comment:
   On the deferrable path this read cannot help and can hurt. Nothing persists an id there, so `_has_stored_external_id` can only return False and never changes the decision. But `TaskStateStoreAccessor.get` raises `AirflowRuntimeError` for every error other than `TASK_STORE_NOT_FOUND` (`task-sdk/src/airflow/sdk/execution_time/context.py:585-587`), and `task_state_store` is unconditionally present in the runtime context on 3.3+ (`task-sdk/src/airflow/sdk/execution_time/task_runner.py:334`).
   
   So a store that errors makes a deferrable retry fail from inside `submit_job` before the task-UUID scan can run, which removes the recovery glue.rst:213-216 promises for that exact path. This arrived with the guard added for my round-2 comment on line 454, so it is worth closing while that change is fresh.
   
   Ask: `and not self.deferrable` on the condition, or hoist the lookup out of `submit_job` and have the deferrable branch pass the decision in.
   
   Related design question, if you would rather fix the cause than the symptom: should the deferrable branch persist the run id before `defer()`? The deferrable path would then reconnect through the cheap store lookup instead of a full `get_job_runs` walk, and this plus the `WAITING`/`STOPPING` gap above would both stop existing there.



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +364,147 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """Record the run id and surface its console link; guarded so one attempt logs it once."""
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first, then falls back to a task-UUID scan. The XCom tier
+        only works on Airflow 2.11-3.2; Airflow 3.3+ clears task XComs before every non-deferral
+        attempt, so it always misses there and the scan runs every time.
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def _has_stored_external_id(self, context: Context) -> bool:
+        task_state_store = context.get("task_state_store")
+        return task_state_store is not None and task_state_store.get(self.external_id_key) is not None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        # Scan only when there's nothing else to go on: first attempt, no store, or a store that
+        # never recorded this key. A stored id (even terminal) means the caller already decided.
+        if self.durable and context["ti"].try_number > 1 and not self._has_stored_external_id(context):
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])

Review Comment:
   Worth a second look now that this runs by default rather than under an opt-in flag. `_find_job_run_id_by_task_uuid` is a `while True` over `get_job_runs(MaxResults=50)` with no page cap and no age window, and the no-match case (the prior attempt died before submitting anything, which is the common retry shape) is the one that walks the job's entire run history.
   
   Two consequences that are new because the flag flipped, not because the function changed:
   
   The scan needs `glue:GetJobRuns`. A task policy granting only `StartJobRun` and `GetJobRun`, which was sufficient before, now hits AccessDenied on every retry.
   
   The failure is swallowed. Both `except Exception` blocks (lines 419 and 434) log at warning and return None, after which the operator submits fresh, producing `ConcurrentRunsExceededException` against the live run, which is the failure this feature exists to prevent. There is no error-level signal to tell an operator why.
   
   Not asking for fail-closed: those blocks are verbatim from the merge base, and turning a transient throttle into a task failure would be worse.
   
   Ask: bound the walk (a page cap, or stop once `StartedOn` predates the DAG run), narrow the `except Exception` to `ClientError`, log the fall-through at error level with the reason, and add `glue:GetJobRuns` to the IAM note in glue.rst.



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -158,6 +203,16 @@ def __init__(
         ),
         **kwargs,
     ):
+        if resume_glue_job_on_retry is not None:
+            # Kept as a real named parameter (not **kwargs) so `default_args` still applies correctly.
+            if AIRFLOW_V_3_3_PLUS:

Review Comment:
   Provider deprecations are scoped by provider version, not Airflow version. The same amazon wheel serves 2.11 through 3.4, and the removal will land in a future amazon release for all of them, so gating the warning on `AIRFLOW_V_3_3_PLUS` hides it from part of the install base for the whole deprecation window.
   
   The usual justification for such a gate does not apply here: below 3.3 the compat stub still sets `self.durable`, and the operator honours it at lines 398 and 447, so `durable` is a functional drop-in for what `resume_glue_job_on_retry` did. The shipped precedent warns unconditionally (`providers/apache/spark/src/airflow/providers/apache/spark/operators/spark_submit.py:405-411`, no version gate).
   
   A team on Airflow 3.1 passing `resume_glue_job_on_retry=True` therefore sees nothing until the parameter is removed, and then gets a `TypeError`.
   
   Ask: drop the gate, invert `test_silent_below_3_3` to assert the warning fires on every supported version, and remove the "on Airflow 3.3 and newer" qualifier at glue.rst:220.



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +364,147 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """Record the run id and surface its console link; guarded so one attempt logs it once."""
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first, then falls back to a task-UUID scan. The XCom tier
+        only works on Airflow 2.11-3.2; Airflow 3.3+ clears task XComs before every non-deferral
+        attempt, so it always misses there and the scan runs every time.
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def _has_stored_external_id(self, context: Context) -> bool:
+        task_state_store = context.get("task_state_store")
+        return task_state_store is not None and task_state_store.get(self.external_id_key) is not None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        # Scan only when there's nothing else to go on: first attempt, no store, or a store that
+        # never recorded this key. A stored id (even terminal) means the caller already decided.
+        if self.durable and context["ti"].try_number > 1 and not self._has_stored_external_id(context):
+            existing_job_run_id = self._find_previous_job_run(context, script_args[self.TASK_UUID_ARG])
+            if existing_job_run_id:
+                self._set_job_run_id(context, existing_job_run_id)
+                return existing_job_run_id
+        self.log.info(
+            "Initializing AWS Glue Job: %s. Wait for completion: %s",
+            self.job_name,
+            self.wait_for_completion,
+        )
+        # A prior get_job_status call may have set this to a stale, terminal run id while checking
+        # whether to reconnect. Clear it so on_kill has nothing to act on if initialize_job raises.
+        self._job_run_id = None
+        glue_job_run = self.hook.initialize_job(script_args, self.run_job_kwargs)
+        # Set before polling so on_kill can stop the run even if the worker dies immediately after.
+        self._set_job_run_id(context, glue_job_run["JobRunId"])
+        # Downstream tasks read this key directly; it's also what feeds the XCom tier above.
+        context["ti"].xcom_push(key="glue_job_run_id", value=glue_job_run["JobRunId"])
+        return glue_job_run["JobRunId"]
+
+    def get_job_status(self, external_id: JsonValue, context: Context) -> str:
+        """Query the raw job run state; a run id Glue no longer knows about degrades to NOT_FOUND."""
+        job_run_id = cast("str", external_id)
+        # This is the first place a reconnecting attempt learns the run id.
+        self._set_job_run_id(context, job_run_id)
+        try:
+            return self.hook.get_job_state(self.job_name, job_run_id)
+        except ClientError as e:
+            if e.response["Error"]["Code"] == "EntityNotFoundException":
+                return NOT_FOUND_STATE
+            raise
+
+    def is_job_active(self, status: str) -> bool:
+        return status not in (*JOB_RUN_TERMINAL_STATES, NOT_FOUND_STATE)
+
+    def is_job_succeeded(self, status: str) -> bool:
+        return status in JOB_RUN_SUCCESS_STATES
+
+    def poll_until_complete(self, external_id: JsonValue, context: Context) -> None:
+        job_run_id = cast("str", external_id)
+        self._set_job_run_id(context, job_run_id)
+        if not self.wait_for_completion:
+            self.log.info("AWS Glue Job: %s. Run Id: %s", self.job_name, job_run_id)
+            return
+        glue_job_run = self.hook.job_completion(
+            self.job_name, job_run_id, self.verbose, self.sleep_before_return
+        )
+        state = glue_job_run["JobRunState"]
+        self.log.info("AWS Glue Job: %s status: %s. Run Id: %s", self.job_name, state, job_run_id)
+        if state not in JOB_RUN_SUCCESS_STATES:

Review Comment:
   Completing the thread from my round-2 comment on glue.rst:158 rather than reversing it. This raise is the fix I asked for, and it is the right end state: the deferrable non-verbose path's `job_complete` waiter already treats `STOPPED` as a failure, so this makes the synchronous path agree with it.
   
   The issue is scoping. The raise lives in `poll_until_complete`, which `execute_resumable` also calls on its non-durable branch, and it fires on first attempts too, so it is not gated on `durable` in any way. Concretely: someone cancels a Glue run in the console, and on main `_handle_state` buckets `STOPPED` in `finished_states` so the task logs `status: STOPPED` and goes green. At this HEAD it goes red and burns a retry that resubmits the job they just cancelled.
   
   The docs do not carry that. The stopped-run paragraph at glue.rst:165-173 sits entirely under "Durable execution", and glue.rst:199 offers `durable=False` as the opt-out, so a reader reasonably concludes `durable=False` restores the old behaviour. It does not.
   
   Ask: move that paragraph out of the durable section and reword it to cover first attempts and `durable=False`, add a sentence saying explicitly that `durable=False` does not restore `STOPPED`-as-success, and drop the "durable" framing from the code comment on the line above.



##########
providers/amazon/src/airflow/providers/amazon/aws/exceptions.py:
##########
@@ -78,3 +78,7 @@ class NeptuneImportTaskCancellationFailedError(AirflowException):
 
 class NeptuneImportTaskFailedError(AirflowException):
     """Raised when a Neptune Analytics import task fails to complete successfully."""
+
+
+class GlueJobRunStoppedError(AirflowException):
+    """Raised when a reconnected Glue job run finishes in a state that is not a real success."""

Review Comment:
   "reconnected" is narrower than the behaviour. `poll_until_complete` raises this on the fresh-submit path as well, and on first attempts, so the docstring understates the blast radius for `durable=False` users.
   
   Ask: drop the word, for example "Raised when a Glue job run finishes in a state that is not a real success."



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +364,147 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """Record the run id and surface its console link; guarded so one attempt logs it once."""
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first, then falls back to a task-UUID scan. The XCom tier
+        only works on Airflow 2.11-3.2; Airflow 3.3+ clears task XComs before every non-deferral
+        attempt, so it always misses there and the scan runs every time.
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):

Review Comment:
   This keeps the merge-base literal `if state in ("RUNNING", "STARTING")` here and again at line 431, but the new `is_job_active` at line 479 is a deny-list that counts `WAITING` and `STOPPING` as active, and the docs added at glue.rst:158 promise the operator reconnects when a run is "waiting for capacity, or being stopped".
   
   This tier matters more than it looks: it is the only recovery mechanism on the deferrable path (`execute` calls `submit_job` directly and nothing persists an id there, so `_has_stored_external_id` is always False), the only one below Airflow 3.3, and the one used on 3.3+ when the worker died before the store write. `WAITING` is exactly "queued behind the concurrent-run limit", which is the situation glue.rst:151-154 builds the whole feature around.
   
   To be fair about severity, this is not a regression: on main the default was `resume_glue_job_on_retry=False`, so attempt 2 resubmitted anyway. The narrower claim is that the implementation does not do what `is_job_active` and its own new docs say.
   
   Ask: `if self.is_job_active(state):` here and `if self.is_job_active(existing_job_run_state):` at 431, plus the history-scan tests parametrized over `WAITING` and `STOPPING` (only the task-state-store tests cover those today).
   
   Different code path from my round-2 comment on line 487, so this is not re-litigating it: that fix landed in `poll_until_complete`, this is the reattach decision.



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +364,147 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """Record the run id and surface its console link; guarded so one attempt logs it once."""
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first, then falls back to a task-UUID scan. The XCom tier

Review Comment:
   Follow-up on my round-2 comment on line 470. The version boundary here is off by three releases, and I should say plainly that my own round-2 wording pointed you at the wrong one.
   
   Pre-attempt XCom clearing shipped in 3.0.0, not 3.3. At tag 3.0.0 the execution API builds `xcom_keys_to_clear` for any attempt with no `next_method`, and the SDK deletes every one of those keys. Present and unconditional at 3.0.0, 3.1.0 and 3.2.0.
   
   Consequences: the XCom tier is dead on every Airflow 3 release, the `if previous_job_run_id:` branch below is dead code there, and glue.rst:180-184 tells 3.0-3.2 users they get a cheap XCom lookup before the scan when in fact every durable retry pays the full paginated history walk. The comment at line 463 ("it's also what feeds the XCom tier above") is false for the same reason.
   
   Ask: reword to "Airflow 2.x only; every Airflow 3 release clears task XComs before each non-deferral attempt", and fix glue.rst:182 to match. Since 2.11 is the only beneficiary, dropping the XCom tier is worth considering. Note also that `test_find_previous_job_run_reuses_from_xcom` and `test_find_previous_job_run_does_not_fall_back_to_scan_on_xcom_state_mismatch` pin a path unreachable on any Airflow 3 release, because they stub `xcom_pull` to return a value the runtime never will.



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -158,6 +203,16 @@ def __init__(
         ),
         **kwargs,
     ):
+        if resume_glue_job_on_retry is not None:
+            # Kept as a real named parameter (not **kwargs) so `default_args` still applies correctly.

Review Comment:
   This comment is about `resume_glue_job_on_retry`, and it is accurate for that parameter, but `durable` itself is not a named parameter on `GlueJobOperator.__init__`. It reaches the mixin through `**kwargs` via `kwargs.setdefault`. `_apply_defaults` only fills parameters present in the decorated function's own signature (`task-sdk/src/airflow/sdk/bases/operator.py:527-529`), and the pre-3.3 fallback stub's `__init__` carries no `_apply_defaults` at all.
   
   Net effect on Airflow 2.11 through 3.2, which is inside this provider's declared floor: `default_args={"durable": False}` leaves `durable` True with no error, while `default_args={"resume_glue_job_on_retry": False}` works. On the versions where the deprecation warning is deliberately suppressed, the deprecated parameter is the only one that works at DAG level. That is not cosmetic there, since `durable` still gates the task-UUID injection and the retry scan below 3.3.
   
   This matters mainly because the PR flips the default to True: the documented way to opt out of a new default does not work on those versions. The skipif reason on `test_default_args_durable_reaches_operator` records the mechanism, but nothing user-facing does.
   
   Ask: `durable: bool | None = None` on `GlueJobOperator.__init__`, forwarded only when set, which preserves the mixin's own default. Fair caveat: the sibling operators route `durable` through `**kwargs` the same way, so if you would rather treat it as a cross-provider issue, then stating the limitation beside the `durable=False` example in glue.rst is the minimum.



##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -139,6 +139,86 @@ To submit a new AWS Glue job you can use :class:`~airflow.providers.amazon.aws.o
   The same AWS IAM role used for the crawler can be used here as well, but it will need
   policies to provide access to the output location for result data.
 
+Durable execution
+==================
+
+``GlueJobOperator`` submits a job run and then polls it to completion on the worker. By default
+the operator runs in a *durable* mode that makes this crash-safe: the Glue job run id is
+persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>` before
+polling begins, so if the worker crashes or is preempted and the task is retried, the operator
+reconnects to the run that is already executing in Glue instead of starting a new one.
+
+This matters more for Glue because a Glue job's ``concurrent_run_limit`` defaults to ``1``, so
+submitting a second run while the first is still active does not create a harmless duplicate, it
+fails outright with ``ConcurrentRunsExceededException`` and the task keeps retrying against a run
+it can never see. Durable execution turns that retry into a normal reconnect.
+
+On retry the operator checks the prior run's state:
+
+* if it is still starting, running, waiting for capacity, or being stopped, the operator
+  reconnects and continues polling
+* if it already succeeded, the operator returns immediately without resubmitting
+* if it failed terminally, or its id has expired and is no longer found, the operator submits the
+  job fresh
+* if it already stopped, the operator submits the job fresh
+
+A stopped run is treated as a failure, not a success. Glue's API has no way to tell a run
+cancelled manually (for example, in the AWS console) apart from one this operator's own
+:meth:`~airflow.providers.amazon.aws.operators.glue.GlueJobOperator.on_kill` stopped, which happens
+whenever ``stop_job_run_on_kill=True`` and the task is killed -- on SIGTERM, on
+``execution_timeout``, or when the task is cleared while running. If the stored state is already
+``STOPPED``, the operator submits fresh. If a reconnect finds the run still stopping and it
+settles into ``STOPPED`` while polling, the operator raises instead of returning a result -- the
+task fails and a normal retry resubmits. Either way, a self-inflicted stop never gets silently
+reported as a false success.
+
+This protection also applies when ``wait_for_completion=False`` -- even though that task attempt
+never polls at all, a retry after a successful submission still reconnects rather than
+resubmitting, since the run id is persisted immediately after submission regardless of whether the
+task waits for it to finish.
+
+Durable execution requires Airflow 3.3 or newer for the task state store lookup above. On earlier
+Airflow versions, or if a prior run was never recorded to the task state store, ``durable=True``
+still recovers a prior run via an older mechanism: the operator checks XCom for a cached run id
+first, then falls back to scanning the job's run history for a run tagged with this task
+instance's identity, and reconnects if it finds one that is still active.
+
+Like the persisted state itself, the stored run id isn't deleted automatically, that only happens
+when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is longer than
+``[state_store] default_retention_days`` (30 days by default) and cleanup runs in between, the run
+id won't be there for the next retry, and the operator falls back to the XCom/scan mechanism
+above rather than reconnecting via task state store. Avoid running cleanup on a schedule shorter
+than your longest ``retry_delay``.
+
+Clearing a task is treated the same as a retry, which matters specifically for a task whose job
+already succeeded: clearing does not delete the stored run id, so the next attempt reads it back
+and returns immediately without submitting anything to Glue. See
+:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
+``[state_store] clear_on_success`` setting that restores "clearing always resubmits."
+
+To opt out and always start a fresh run on retry, set ``durable=False``:
+
+.. code-block:: python
+
+  glue_job = GlueJobOperator(
+      task_id="glue_job",
+      job_name="my_glue_job",
+      script_location="s3://glue-examples/glue-scripts/sample_aws_glue_job.py",
+      durable=False,
+  )
+
+The task state store lookup above is only used on the synchronous path -- when ``deferrable=True``
+is set, the Triggerer already tracks the run across the wait, so a run id is never persisted there.
+``durable`` still has an effect on retry, though: a retry of a deferrable task resubmits by

Review Comment:
   This sentence is now backwards. With `durable=True` as the default, a retry of a deferrable task scans for the tagged run first; resubmitting is what you get when you opt out with `durable=False`. As written, "resubmits by default" asserts the opposite of what line 447 of the operator does.
   
   Ask: "would otherwise resubmit". (Separate sentence from the deferrable point in my round-2 comment, so this is a distinct fix.)



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +364,147 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """Record the run id and surface its console link; guarded so one attempt logs it once."""
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:

Review Comment:
   With `durable` defaulting to True, this injects `--airflow_task_uuid` into the Arguments of every Glue job run. `initialize_job` passes the dict straight to `start_job_run` (`providers/amazon/src/airflow/providers/amazon/aws/hooks/glue.py:271`), and Glue surfaces Arguments to the script as command-line arguments. At the merge base this happened only under `resume_glue_job_on_retry=True`, which defaulted to False, so a provider upgrade alone now changes the argv of unchanged production scripts.
   
   Hedging this deliberately, because it is not clear-cut: `getResolvedOptions`, the documented Glue idiom, uses `parse_known_args` and tolerates the extra argument, so scripts following the docs are unaffected. A script doing its own strict parsing with `argparse.ArgumentParser().parse_args()` exits 2 on an unrecognised argument. Common, not universal.
   
   Ask: either call this out in the docs as an upgrade note, or inject only where the tag is actually the recovery mechanism (`self.deferrable or not AIRFLOW_V_3_3_PLUS`), since on 3.3+ synchronous runs the task state store is the primary tier and the tag only covers the crash-before-persist window the mixin's docstring already declares unclosable.



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -35,17 +38,52 @@
 )
 from airflow.providers.amazon.aws.utils import validate_execute_complete_event
 from airflow.providers.amazon.aws.utils.mixins import aws_template_fields
+from airflow.providers.amazon.version_compat import AIRFLOW_V_3_3_PLUS
 from airflow.providers.common.compat.openlineage.utils.spark import (
     inject_parent_job_information_into_glue_arguments,
     inject_transport_information_into_glue_arguments,
 )
 from airflow.providers.common.compat.sdk import AirflowException, conf
 
+# ResumableJobMixin only exists on Airflow 3; this provider still targets >=2.11. Drop this
+# fallback once the provider's minimum Airflow version is >=3.0.

Review Comment:
   Threshold looks wrong. The mixin reads `context["task_state_store"]`, which only exists on Airflow 3.3+, and the stub's own docstring below, the operator docstring at line 186, and the test class gate all say 3.3. A maintainer who bumps the floor to 3.0 and follows this comment would delete the fallback and break every Glue DAG on 3.0 through 3.2 with a parse-time ImportError.
   
   Ask: `>=3.3`. Worth noting none of the four shipped sibling stubs (redshift_data, databricks, snowflake, bigquery) carries this comment at all.



##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -139,6 +139,86 @@ To submit a new AWS Glue job you can use :class:`~airflow.providers.amazon.aws.o
   The same AWS IAM role used for the crawler can be used here as well, but it will need
   policies to provide access to the output location for result data.
 
+Durable execution
+==================
+
+``GlueJobOperator`` submits a job run and then polls it to completion on the worker. By default
+the operator runs in a *durable* mode that makes this crash-safe: the Glue job run id is
+persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>` before
+polling begins, so if the worker crashes or is preempted and the task is retried, the operator
+reconnects to the run that is already executing in Glue instead of starting a new one.
+
+This matters more for Glue because a Glue job's ``concurrent_run_limit`` defaults to ``1``, so
+submitting a second run while the first is still active does not create a harmless duplicate, it
+fails outright with ``ConcurrentRunsExceededException`` and the task keeps retrying against a run
+it can never see. Durable execution turns that retry into a normal reconnect.
+
+On retry the operator checks the prior run's state:
+
+* if it is still starting, running, waiting for capacity, or being stopped, the operator
+  reconnects and continues polling
+* if it already succeeded, the operator returns immediately without resubmitting
+* if it failed terminally, or its id has expired and is no longer found, the operator submits the
+  job fresh
+* if it already stopped, the operator submits the job fresh
+
+A stopped run is treated as a failure, not a success. Glue's API has no way to tell a run
+cancelled manually (for example, in the AWS console) apart from one this operator's own
+:meth:`~airflow.providers.amazon.aws.operators.glue.GlueJobOperator.on_kill` stopped, which happens
+whenever ``stop_job_run_on_kill=True`` and the task is killed -- on SIGTERM, on
+``execution_timeout``, or when the task is cleared while running. If the stored state is already
+``STOPPED``, the operator submits fresh. If a reconnect finds the run still stopping and it
+settles into ``STOPPED`` while polling, the operator raises instead of returning a result -- the
+task fails and a normal retry resubmits. Either way, a self-inflicted stop never gets silently
+reported as a false success.
+
+This protection also applies when ``wait_for_completion=False`` -- even though that task attempt
+never polls at all, a retry after a successful submission still reconnects rather than
+resubmitting, since the run id is persisted immediately after submission regardless of whether the
+task waits for it to finish.
+
+Durable execution requires Airflow 3.3 or newer for the task state store lookup above. On earlier
+Airflow versions, or if a prior run was never recorded to the task state store, ``durable=True``
+still recovers a prior run via an older mechanism: the operator checks XCom for a cached run id
+first, then falls back to scanning the job's run history for a run tagged with this task
+instance's identity, and reconnects if it finds one that is still active.
+
+Like the persisted state itself, the stored run id isn't deleted automatically, that only happens

Review Comment:
   `airflow db clean` also targets this table: `_TableConfig(table_name="task_state_store", recency_column_name="expires_at", ...)` in `airflow-core/src/airflow/utils/db_cleanup.py`. So "that only happens when someone runs `airflow state-store clean`" understates when the stored id can disappear.
   
   Worth fixing because this paragraph's own mitigation advice, avoiding a cleanup schedule shorter than the longest `retry_delay`, only lands if the reader knows `db clean` counts too.



##########
providers/amazon/tests/unit/amazon/aws/operators/test_glue.py:
##########
@@ -163,6 +171,62 @@ def test_execute_deferrable(self, _, mock_initialize_job):
         assert defer.value.trigger.attempts == 75
         assert defer.value.trigger.aws_conn_id == "aws_default"
 
+    @mock.patch.object(GlueJobHook, "initialize_job")
+    @mock.patch.object(GlueJobHook, "get_conn")
+    def test_deferrable_first_attempt_injects_task_uuid_but_skips_scan(

Review Comment:
   The test asserts the `TASK_UUID_ARG` half of its name but nothing for "skips_scan", so the part most likely to regress is unguarded. `test_first_attempt_skips_the_retry_lookup_entirely` at line 1097 already does this properly.
   
   Ask: add `mock_get_conn.return_value.get_job_runs.assert_not_called()`. Worth more than a naming nit here, because the scan has no page cap and an unspecced conn mock returns a truthy `NextToken`, so if the `try_number > 1` gate is ever dropped this test hangs rather than fails.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3765043941


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -384,6 +364,147 @@ def on_kill(self):
             if not response["SuccessfulSubmissions"]:
                 self.log.error("Failed to stop AWS Glue Job: %s. Run Id: %s", self.job_name, self._job_run_id)
 
+    def _set_job_run_id(self, context: Context, job_run_id: str) -> None:
+        """Record the run id and surface its console link; guarded so one attempt logs it once."""
+        if self._job_run_id == job_run_id:
+            return
+        self._job_run_id = job_run_id
+        GlueJobRunDetailsLink.persist(
+            context=context,
+            operator=self,
+            region_name=self.hook.conn_region_name,
+            aws_partition=self.hook.conn_partition,
+            job_name=urllib.parse.quote(self.job_name, safe=""),
+            job_run_id=job_run_id,
+        )
+        self.log.info(
+            "You can monitor this Glue Job run at: %s",
+            GlueJobRunDetailsLink.format_str.format(
+                aws_domain=GlueJobRunDetailsLink.get_aws_domain(self.hook.conn_partition),
+                region_name=self.hook.conn_region_name,
+                job_name=urllib.parse.quote(self.job_name, safe=""),
+                job_run_id=job_run_id,
+            ),
+        )
+
+    def _build_script_args(self, context: Context) -> dict:
+        script_args = dict(self.script_args)
+        if self.openlineage_inject_parent_job_info:
+            self.log.info("Injecting OpenLineage parent job information into Glue job arguments.")
+            script_args = inject_parent_job_information_into_glue_arguments(script_args, context)
+        if self.openlineage_inject_transport_info:
+            self.log.info("Injecting OpenLineage transport information into Glue job arguments.")
+            script_args = inject_transport_information_into_glue_arguments(script_args, context)
+        if self.durable:
+            script_args, _ = self._prepare_script_args_with_task_uuid(context, base_args=script_args)
+        return script_args
+
+    def _find_previous_job_run(self, context: Context, task_uuid: str) -> str | None:
+        """
+        Look for a Glue job run this task instance already started.
+
+        Checks XCom for a cached run id first, then falls back to a task-UUID scan. The XCom tier
+        only works on Airflow 2.11-3.2; Airflow 3.3+ clears task XComs before every non-deferral
+        attempt, so it always misses there and the scan runs every time.
+        """
+        ti = context["ti"]
+        previous_job_run_id = ti.xcom_pull(key="glue_job_run_id", task_ids=ti.task_id)
+        if previous_job_run_id:
+            try:
+                job_run = self.hook.conn.get_job_run(JobName=self.job_name, RunId=previous_job_run_id)
+                state = job_run.get("JobRun", {}).get("JobRunState")
+                self.log.info("Previous Glue job_run_id: %s, state: %s", previous_job_run_id, state)
+                if state in ("RUNNING", "STARTING"):
+                    return previous_job_run_id
+            except Exception:
+                self.log.warning("Failed to get previous Glue job run state", exc_info=True)
+        else:
+            try:
+                existing = self._find_job_run_id_by_task_uuid(task_uuid)
+                if existing:
+                    existing_job_run_id, existing_job_run_state = existing
+                    self.log.info(
+                        "Found Glue job_run_id by task UUID: %s, state: %s",
+                        existing_job_run_id,
+                        existing_job_run_state,
+                    )
+                    if existing_job_run_state in ("RUNNING", "STARTING"):
+                        ti.xcom_push(key="glue_job_run_id", value=existing_job_run_id)
+                        return existing_job_run_id
+            except Exception:
+                self.log.warning("Failed to find previous Glue job run by task UUID", exc_info=True)
+        return None
+
+    def _has_stored_external_id(self, context: Context) -> bool:
+        task_state_store = context.get("task_state_store")
+        return task_state_store is not None and task_state_store.get(self.external_id_key) is not None
+
+    def submit_job(self, context: Context) -> str:
+        """Start a Glue job run and return its run id, or reconnect to one this task already started."""
+        script_args = self._build_script_args(context)
+        # Scan only when there's nothing else to go on: first attempt, no store, or a store that
+        # never recorded this key. A stored id (even terminal) means the caller already decided.
+        if self.durable and context["ti"].try_number > 1 and not self._has_stored_external_id(context):

Review Comment:
   Fixed, the store check is now bypassed specifically on the deferrable path (`self.deferrable or not self._has_stored_external_id(...)`), not the whole scan condition, so a store read error there no longer blocks reattachment. Logged the persist beforedefer redesign as a follow-up rather than building it now, it needs the mixin's reconnect decision step exposed separately from its polling, which is a task-sdk change, not a quick fix here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #71211:
URL: https://github.com/apache/airflow/pull/71211#discussion_r3772644493


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -158,6 +203,16 @@ def __init__(
         ),
         **kwargs,
     ):
+        if resume_glue_job_on_retry is not None:
+            # Kept as a real named parameter (not **kwargs) so `default_args` still applies correctly.
+            if AIRFLOW_V_3_3_PLUS:

Review Comment:
   Following up since the design moved since this was marked resolved. 
   
   The warning still fires unconditionally on every version, that part has not changed. I realised from user lens and `durable` set directly no longer does anything at all below 3.3 (previously a functional drop-in, which was the premise here, which didnt feel right because old users should not be aware of this flag), only `resume_glue_job_on_retry` activates the old XCom/scan mechanism there now. 
   
   So the message itself is version-aware, below 3.3 it says "On Airflow 3.3+, use durable instead" instead of telling that user to switch to something currently inert for them and glue.rst picked up the same nuance for the same reason. The warning and its removal timeline (tied to the provider's minimum Airflow version, not the caller's) still apply to every version, per your original ask.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Replace `resume_glue_job_on_retry` with durable execution for `GlueJobOperator` [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh merged PR #71211:
URL: https://github.com/apache/airflow/pull/71211


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org