You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "yash1thsa (via GitHub)" <gi...@apache.org> on 2025/09/16 01:40:36 UTC

[PR] Fix(4561) - AWS BatchOperator does not fetch log entries for deferred jobs [airflow]

yash1thsa opened a new pull request, #55703:
URL: https://github.com/apache/airflow/pull/55703

   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #54561 
   
   **Issue**: 
   AWS BatchOperator does not fetch log entries for deferred jobs. The issue is prevelant in MWAA and other deployment as well. 
   
   **Steps**:
   - Reproduced the issue in local with AWS connection  
   - Code: For deferred path in batch operator module, add monitor_job function as a downstream step
   - Tested the fix with both test cases and running in local with AWS connection
   
   **Tests**:
   - Happy path with awslog_enabled set to True
   - Happy path with awslog_enabled set to False
   - Failure step for validation error
   
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix(4561) - AWS BatchOperator does not fetch log entries for deferred jobs [airflow]

Posted by "yash1thsa (via GitHub)" <gi...@apache.org>.
yash1thsa commented on code in PR #55703:
URL: https://github.com/apache/airflow/pull/55703#discussion_r2353160368


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py:
##########
@@ -250,14 +250,22 @@ def execute(self, context: Context) -> str | None:
 
         return self.job_id
 
-    def execute_complete(self, context: Context, event: dict[str, Any] | None = None) -> str:
+    def execute_complete(
+        self, context: Context, event: dict[str, Any] | None = None
+    ) -> str:
         validated_event = validate_execute_complete_event(event)
 
         if validated_event["status"] != "success":
             raise AirflowException(f"Error while running job: {validated_event}")
 
-        self.log.info("Job completed.")
-        return validated_event["job_id"]
+        self.job_id = validated_event["job_id"]
+
+        # Fetch logs if awslogs_enabled
+        if self.awslogs_enabled:
+            self.monitor_job(context)  # fetch logs, no need to return

Review Comment:
   @o-nikolas  Thanks for the comment - self.monitor_job should be triggered only once through execute_complete instead of execute method. As a result users should not be seeing multiple start up comments. Here is a formated version of the log , let me know your thoughts (looks similar if deferable is set to false) - 
   
   Log message source details
   sources = [
     "/root/airflow/logs/dag_id=batch_deferred_log/run_id=manual__2025-09-16T16:39:34.512871+00:00/task_id=run_batch_job/attempt=1.log",
     "/root/airflow/logs/dag_id=batch_deferred_log/run_id=manual__2025-09-16T16:39:34.512871+00:00/task_id=run_batch_job/attempt=1.log.trigger.3.log"
   ]
   
   INFO     DAG bundles loaded: dags-folder
            source=airflow.dag_processing.bundles.manager.DagBundlesManager
            loc=manager.py:179
   
   INFO     Filling up the DagBag from /files/dags/batch_deferred_log.py
            source=airflow.models.dagbag.DagBag
            loc=dagbag.py:593
   
   INFO     Running AWS Batch job - job definition: my-test-definition - on queue my-test-queue
            source=airflow.task.operators.airflow.providers.amazon.aws.operators.batch.BatchOperator
            loc=batch.py:280
   
   INFO     AWS Connection (conn_id='aws_default1', conn_type='aws') credentials retrieved from login and password.
            source=airflow.providers.amazon.aws.utils.connection_wrapper.AwsConnectionWrapper
            loc=connection_wrapper.py:334
   
   INFO     AWS Batch job (d3038fd5-5982-4d1a-bfc3-57a6fe256c7d) started
            details={ ResponseMetadata:{...}, jobArn:..., jobName:..., jobId:... }
            source=airflow.task.operators.airflow.providers.amazon.aws.operators.batch.BatchOperator
            loc=batch.py:327
   
   INFO     Pausing task as DEFERRED
            dag_id=batch_deferred_log
            task_id=run_batch_job
            run_id=manual__2025-09-16T16:39:34.512871+00:00
            source=task
            loc=task_runner.py:844
   
   WARNING  No XCom value found; defaulting to None
            key=batch_job_definition
            dag_id=batch_deferred_log task_id=run_batch_job run_id=manual__2025-09-16T16:39:34.512871+00:00 map_index=-1
            source=task
            loc=xcom.py:264
   
   WARNING  No XCom value found; defaulting to None
            key=batch_job_queue
            dag_id=batch_deferred_log task_id=run_batch_job run_id=manual__2025-09-16T16:39:34.512871+00:00 map_index=-1
            source=task
            loc=xcom.py:264
   
   WARNING  No XCom value found; defaulting to None
            key=cloudwatch_events
            dag_id=batch_deferred_log task_id=run_batch_job run_id=manual__2025-09-16T16:39:34.512871+00:00 map_index=-1
            source=task
            loc=xcom.py:264
   
   INFO     trigger batch_deferred_log/manual__2025-09-16T16:39:34.512871+00:00/run_batch_job/-1/1 (ID 2) starting
            loc=triggerer_job_runner.py:1099
   
   INFO     AWS Connection (conn_id='aws_default1', conn_type='aws') credentials retrieved from login and password.
            source=airflow.providers.amazon.aws.utils.connection_wrapper.AwsConnectionWrapper
            loc=connection_wrapper.py:334
   
   INFO     Batch job d3038fd5-5982-4d1a-bfc3-57a6fe256c7d not ready yet: ['STARTING']
            source=airflow.providers.amazon.aws.utils.waiter_with_logging
            loc=waiter_with_logging.py:158
   
   INFO     Batch job d3038fd5-5982-4d1a-bfc3-57a6fe256c7d not ready yet: ['RUNNING']
            source=airflow.providers.amazon.aws.utils.waiter_with_logging
            loc=waiter_with_logging.py:158
   
   INFO     Trigger fired
            event name=batch_deferred_log/manual__2025-09-16T16:39:34.512871+00:00/run_batch_job/-1/1 (ID 2)
            result=TriggerEvent<{'status': 'success', 'job_id': 'd3038fd5-5982-4d1a-bfc3-57a6fe256c7d'}>
            loc=triggerer_job_runner.py:1102
   
   INFO     trigger completed
            name=batch_deferred_log/manual__2025-09-16T16:39:34.512871+00:00/run_batch_job/-1/1 (ID 2)
            loc=triggerer_job_runner.py:1123
   
   INFO     DAG bundles loaded: dags-folder
            source=airflow.dag_processing.bundles.manager.DagBundlesManager
            loc=manager.py:179
   
   INFO     Filling up the DagBag from /files/dags/batch_deferred_log.py
            source=airflow.models.dagbag.DagBag
            loc=dagbag.py:593
   
   INFO     AWS Connection (conn_id='aws_default1', conn_type='aws') credentials retrieved from login and password.
            source=airflow.providers.amazon.aws.utils.connection_wrapper.AwsConnectionWrapper
            loc=connection_wrapper.py:334
   
   INFO     AWS Batch job (d3038fd5-5982-4d1a-bfc3-57a6fe256c7d) Job Definition ARN: ...
            Job Queue ARN: ...
            source=airflow.task.operators.airflow.providers.amazon.aws.operators.batch.BatchOperator
            loc=batch.py:350
   
   INFO     AWS Batch job (...) check status (SUCCEEDED) in ['RUNNING','SUCCEEDED','FAILED']
            source=airflow.task.hooks.airflow.providers.amazon.aws.hooks.batch_client.BatchClientHook
            loc=batch_client.py:381
   
   INFO     AWS Batch job (...) check status (SUCCEEDED) in ['SUCCEEDED','FAILED']
            source=airflow.task.hooks.airflow.providers.amazon.aws.hooks.batch_client.BatchClientHook
            loc=batch_client.py:381
   
   INFO     AWS Connection (conn_id='aws_default1', conn_type='aws') credentials retrieved from login and password.
            source=airflow.providers.amazon.aws.utils.connection_wrapper.AwsConnectionWrapper
            loc=connection_wrapper.py:334
   
   INFO     [2025-09-16 16:39:58,113] hello world
            source=airflow.task.operators.airflow.providers.amazon.aws.operators.batch.BatchOperator
            loc=task_log_fetcher.py:75
   
   INFO     AWS Batch job (...) has completed
            source=airflow.task.hooks.airflow.providers.amazon.aws.hooks.batch_client.BatchClientHook
            loc=batch_client.py:311
   
   INFO     AWS Batch job (...) CloudWatch Events details found. Links to logs:
            source=airflow.task.operators.airflow.providers.amazon.aws.operators.batch.BatchOperator
            loc=batch.py:392
   
   INFO     (empty log line)
            source=airflow.task.operators.airflow.providers.amazon.aws.operators.batch.BatchOperator
            loc=batch.py:395
   
   INFO     AWS Batch job (...) succeeded
            details={ jobArn:..., jobName:..., jobId:..., jobQueue:..., status:..., attempts:[...], container:{...}, etc. }
            source=airflow.task.hooks.airflow.providers.amazon.aws.hooks.batch_client.BatchClientHook
            loc=batch_client.py:270
   
   INFO     AWS Batch job (...) succeeded
            source=airflow.task.operators.airflow.providers.amazon.aws.operators.batch.BatchOperator
            loc=batch.py:411
   
   INFO     Job completed successfully for job_id: d3038fd5-5982-4d1a-bfc3-57a6fe256c7d
            source=airflow.task.operators.airflow.providers.amazon.aws.operators.batch.BatchOperator
            loc=batch.py:267
   
   INFO     Pushing xcom
            ti=RuntimeTaskInstance(...full object...)
            source=task
            loc=task_runner.py:1352
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix(4561) - AWS BatchOperator does not fetch log entries for deferred jobs [airflow]

Posted by "yash1thsa (via GitHub)" <gi...@apache.org>.
yash1thsa commented on PR #55703:
URL: https://github.com/apache/airflow/pull/55703#issuecomment-3300491664

   Cleaned up all CI errors and rebased to master. Please review and let me know for any issues


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix(4561) - AWS BatchOperator does not fetch log entries for deferred jobs [airflow]

Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #55703:
URL: https://github.com/apache/airflow/pull/55703#issuecomment-3298860471

   Static checks are failing, please fix them. If you're not familiar, you can read the [doc](https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst) about it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix(4561) - AWS BatchOperator does not fetch log entries for deferred jobs [airflow]

Posted by "yash1thsa (via GitHub)" <gi...@apache.org>.
yash1thsa commented on PR #55703:
URL: https://github.com/apache/airflow/pull/55703#issuecomment-3299667559

   > Static checks are failing, please fix them. If you're not familiar, you can read the [doc](https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst) about it.
   
   @vincbeck  Thanks for document. Let me review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix(4561) - AWS BatchOperator does not fetch log entries for deferred jobs [airflow]

Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas merged PR #55703:
URL: https://github.com/apache/airflow/pull/55703


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix(4561) - AWS BatchOperator does not fetch log entries for deferred jobs [airflow]

Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #55703:
URL: https://github.com/apache/airflow/pull/55703#discussion_r2356127415


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py:
##########
@@ -250,14 +250,22 @@ def execute(self, context: Context) -> str | None:
 
         return self.job_id
 
-    def execute_complete(self, context: Context, event: dict[str, Any] | None = None) -> str:
+    def execute_complete(
+        self, context: Context, event: dict[str, Any] | None = None
+    ) -> str:
         validated_event = validate_execute_complete_event(event)
 
         if validated_event["status"] != "success":
             raise AirflowException(f"Error while running job: {validated_event}")
 
-        self.log.info("Job completed.")
-        return validated_event["job_id"]
+        self.job_id = validated_event["job_id"]
+
+        # Fetch logs if awslogs_enabled
+        if self.awslogs_enabled:
+            self.monitor_job(context)  # fetch logs, no need to return

Review Comment:
   Thanks for the log output! Looks good :) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix(4561) - AWS BatchOperator does not fetch log entries for deferred jobs [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #55703:
URL: https://github.com/apache/airflow/pull/55703#issuecomment-3303828572

   Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix(4561) - AWS BatchOperator does not fetch log entries for deferred jobs [airflow]

Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #55703:
URL: https://github.com/apache/airflow/pull/55703#discussion_r2352929375


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py:
##########
@@ -250,14 +250,22 @@ def execute(self, context: Context) -> str | None:
 
         return self.job_id
 
-    def execute_complete(self, context: Context, event: dict[str, Any] | None = None) -> str:
+    def execute_complete(
+        self, context: Context, event: dict[str, Any] | None = None
+    ) -> str:
         validated_event = validate_execute_complete_event(event)
 
         if validated_event["status"] != "success":
             raise AirflowException(f"Error while running job: {validated_event}")
 
-        self.log.info("Job completed.")
-        return validated_event["job_id"]
+        self.job_id = validated_event["job_id"]
+
+        # Fetch logs if awslogs_enabled
+        if self.awslogs_enabled:
+            self.monitor_job(context)  # fetch logs, no need to return

Review Comment:
   We know the job is done by this point. Is `self.monitor_job` going to log anything about starting to monitor again for a second time that will confuse the users?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org