You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "yash1thsa (via GitHub)" <gi...@apache.org> on 2025/09/16 01:40:36 UTC
[PR] Fix(4561) - AWS BatchOperator does not fetch log entries for deferred jobs [airflow]
yash1thsa opened a new pull request, #55703:
URL: https://github.com/apache/airflow/pull/55703
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #54561
**Issue**:
AWS BatchOperator does not fetch log entries for deferred jobs. The issue is prevelant in MWAA and other deployment as well.
**Steps**:
- Reproduced the issue in local with AWS connection
- Code: For deferred path in batch operator module, add monitor_job function as a downstream step
- Tested the fix with both test cases and running in local with AWS connection
**Tests**:
- Happy path with awslog_enabled set to True
- Happy path with awslog_enabled set to False
- Failure step for validation error
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix(4561) - AWS BatchOperator does not fetch log entries for deferred jobs [airflow]
Posted by "yash1thsa (via GitHub)" <gi...@apache.org>.
yash1thsa commented on code in PR #55703:
URL: https://github.com/apache/airflow/pull/55703#discussion_r2353160368
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py:
##########
@@ -250,14 +250,22 @@ def execute(self, context: Context) -> str | None:
return self.job_id
- def execute_complete(self, context: Context, event: dict[str, Any] | None = None) -> str:
+ def execute_complete(
+ self, context: Context, event: dict[str, Any] | None = None
+ ) -> str:
validated_event = validate_execute_complete_event(event)
if validated_event["status"] != "success":
raise AirflowException(f"Error while running job: {validated_event}")
- self.log.info("Job completed.")
- return validated_event["job_id"]
+ self.job_id = validated_event["job_id"]
+
+ # Fetch logs if awslogs_enabled
+ if self.awslogs_enabled:
+ self.monitor_job(context) # fetch logs, no need to return
Review Comment:
@o-nikolas Thanks for the comment - self.monitor_job should be triggered only once through execute_complete instead of execute method. As a result users should not be seeing multiple start up comments. Here is a formated version of the log , let me know your thoughts (looks similar if deferable is set to false) -
Log message source details
sources = [
"/root/airflow/logs/dag_id=batch_deferred_log/run_id=manual__2025-09-16T16:39:34.512871+00:00/task_id=run_batch_job/attempt=1.log",
"/root/airflow/logs/dag_id=batch_deferred_log/run_id=manual__2025-09-16T16:39:34.512871+00:00/task_id=run_batch_job/attempt=1.log.trigger.3.log"
]
INFO DAG bundles loaded: dags-folder
source=airflow.dag_processing.bundles.manager.DagBundlesManager
loc=manager.py:179
INFO Filling up the DagBag from /files/dags/batch_deferred_log.py
source=airflow.models.dagbag.DagBag
loc=dagbag.py:593
INFO Running AWS Batch job - job definition: my-test-definition - on queue my-test-queue
source=airflow.task.operators.airflow.providers.amazon.aws.operators.batch.BatchOperator
loc=batch.py:280
INFO AWS Connection (conn_id='aws_default1', conn_type='aws') credentials retrieved from login and password.
source=airflow.providers.amazon.aws.utils.connection_wrapper.AwsConnectionWrapper
loc=connection_wrapper.py:334
INFO AWS Batch job (d3038fd5-5982-4d1a-bfc3-57a6fe256c7d) started
details={ ResponseMetadata:{...}, jobArn:..., jobName:..., jobId:... }
source=airflow.task.operators.airflow.providers.amazon.aws.operators.batch.BatchOperator
loc=batch.py:327
INFO Pausing task as DEFERRED
dag_id=batch_deferred_log
task_id=run_batch_job
run_id=manual__2025-09-16T16:39:34.512871+00:00
source=task
loc=task_runner.py:844
WARNING No XCom value found; defaulting to None
key=batch_job_definition
dag_id=batch_deferred_log task_id=run_batch_job run_id=manual__2025-09-16T16:39:34.512871+00:00 map_index=-1
source=task
loc=xcom.py:264
WARNING No XCom value found; defaulting to None
key=batch_job_queue
dag_id=batch_deferred_log task_id=run_batch_job run_id=manual__2025-09-16T16:39:34.512871+00:00 map_index=-1
source=task
loc=xcom.py:264
WARNING No XCom value found; defaulting to None
key=cloudwatch_events
dag_id=batch_deferred_log task_id=run_batch_job run_id=manual__2025-09-16T16:39:34.512871+00:00 map_index=-1
source=task
loc=xcom.py:264
INFO trigger batch_deferred_log/manual__2025-09-16T16:39:34.512871+00:00/run_batch_job/-1/1 (ID 2) starting
loc=triggerer_job_runner.py:1099
INFO AWS Connection (conn_id='aws_default1', conn_type='aws') credentials retrieved from login and password.
source=airflow.providers.amazon.aws.utils.connection_wrapper.AwsConnectionWrapper
loc=connection_wrapper.py:334
INFO Batch job d3038fd5-5982-4d1a-bfc3-57a6fe256c7d not ready yet: ['STARTING']
source=airflow.providers.amazon.aws.utils.waiter_with_logging
loc=waiter_with_logging.py:158
INFO Batch job d3038fd5-5982-4d1a-bfc3-57a6fe256c7d not ready yet: ['RUNNING']
source=airflow.providers.amazon.aws.utils.waiter_with_logging
loc=waiter_with_logging.py:158
INFO Trigger fired
event name=batch_deferred_log/manual__2025-09-16T16:39:34.512871+00:00/run_batch_job/-1/1 (ID 2)
result=TriggerEvent<{'status': 'success', 'job_id': 'd3038fd5-5982-4d1a-bfc3-57a6fe256c7d'}>
loc=triggerer_job_runner.py:1102
INFO trigger completed
name=batch_deferred_log/manual__2025-09-16T16:39:34.512871+00:00/run_batch_job/-1/1 (ID 2)
loc=triggerer_job_runner.py:1123
INFO DAG bundles loaded: dags-folder
source=airflow.dag_processing.bundles.manager.DagBundlesManager
loc=manager.py:179
INFO Filling up the DagBag from /files/dags/batch_deferred_log.py
source=airflow.models.dagbag.DagBag
loc=dagbag.py:593
INFO AWS Connection (conn_id='aws_default1', conn_type='aws') credentials retrieved from login and password.
source=airflow.providers.amazon.aws.utils.connection_wrapper.AwsConnectionWrapper
loc=connection_wrapper.py:334
INFO AWS Batch job (d3038fd5-5982-4d1a-bfc3-57a6fe256c7d) Job Definition ARN: ...
Job Queue ARN: ...
source=airflow.task.operators.airflow.providers.amazon.aws.operators.batch.BatchOperator
loc=batch.py:350
INFO AWS Batch job (...) check status (SUCCEEDED) in ['RUNNING','SUCCEEDED','FAILED']
source=airflow.task.hooks.airflow.providers.amazon.aws.hooks.batch_client.BatchClientHook
loc=batch_client.py:381
INFO AWS Batch job (...) check status (SUCCEEDED) in ['SUCCEEDED','FAILED']
source=airflow.task.hooks.airflow.providers.amazon.aws.hooks.batch_client.BatchClientHook
loc=batch_client.py:381
INFO AWS Connection (conn_id='aws_default1', conn_type='aws') credentials retrieved from login and password.
source=airflow.providers.amazon.aws.utils.connection_wrapper.AwsConnectionWrapper
loc=connection_wrapper.py:334
INFO [2025-09-16 16:39:58,113] hello world
source=airflow.task.operators.airflow.providers.amazon.aws.operators.batch.BatchOperator
loc=task_log_fetcher.py:75
INFO AWS Batch job (...) has completed
source=airflow.task.hooks.airflow.providers.amazon.aws.hooks.batch_client.BatchClientHook
loc=batch_client.py:311
INFO AWS Batch job (...) CloudWatch Events details found. Links to logs:
source=airflow.task.operators.airflow.providers.amazon.aws.operators.batch.BatchOperator
loc=batch.py:392
INFO (empty log line)
source=airflow.task.operators.airflow.providers.amazon.aws.operators.batch.BatchOperator
loc=batch.py:395
INFO AWS Batch job (...) succeeded
details={ jobArn:..., jobName:..., jobId:..., jobQueue:..., status:..., attempts:[...], container:{...}, etc. }
source=airflow.task.hooks.airflow.providers.amazon.aws.hooks.batch_client.BatchClientHook
loc=batch_client.py:270
INFO AWS Batch job (...) succeeded
source=airflow.task.operators.airflow.providers.amazon.aws.operators.batch.BatchOperator
loc=batch.py:411
INFO Job completed successfully for job_id: d3038fd5-5982-4d1a-bfc3-57a6fe256c7d
source=airflow.task.operators.airflow.providers.amazon.aws.operators.batch.BatchOperator
loc=batch.py:267
INFO Pushing xcom
ti=RuntimeTaskInstance(...full object...)
source=task
loc=task_runner.py:1352
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix(4561) - AWS BatchOperator does not fetch log entries for deferred jobs [airflow]
Posted by "yash1thsa (via GitHub)" <gi...@apache.org>.
yash1thsa commented on PR #55703:
URL: https://github.com/apache/airflow/pull/55703#issuecomment-3300491664
Cleaned up all CI errors and rebased to master. Please review and let me know for any issues
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix(4561) - AWS BatchOperator does not fetch log entries for deferred jobs [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #55703:
URL: https://github.com/apache/airflow/pull/55703#issuecomment-3298860471
Static checks are failing, please fix them. If you're not familiar, you can read the [doc](https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst) about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix(4561) - AWS BatchOperator does not fetch log entries for deferred jobs [airflow]
Posted by "yash1thsa (via GitHub)" <gi...@apache.org>.
yash1thsa commented on PR #55703:
URL: https://github.com/apache/airflow/pull/55703#issuecomment-3299667559
> Static checks are failing, please fix them. If you're not familiar, you can read the [doc](https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst) about it.
@vincbeck Thanks for document. Let me review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix(4561) - AWS BatchOperator does not fetch log entries for deferred jobs [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas merged PR #55703:
URL: https://github.com/apache/airflow/pull/55703
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix(4561) - AWS BatchOperator does not fetch log entries for deferred jobs [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #55703:
URL: https://github.com/apache/airflow/pull/55703#discussion_r2356127415
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py:
##########
@@ -250,14 +250,22 @@ def execute(self, context: Context) -> str | None:
return self.job_id
- def execute_complete(self, context: Context, event: dict[str, Any] | None = None) -> str:
+ def execute_complete(
+ self, context: Context, event: dict[str, Any] | None = None
+ ) -> str:
validated_event = validate_execute_complete_event(event)
if validated_event["status"] != "success":
raise AirflowException(f"Error while running job: {validated_event}")
- self.log.info("Job completed.")
- return validated_event["job_id"]
+ self.job_id = validated_event["job_id"]
+
+ # Fetch logs if awslogs_enabled
+ if self.awslogs_enabled:
+ self.monitor_job(context) # fetch logs, no need to return
Review Comment:
Thanks for the log output! Looks good :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix(4561) - AWS BatchOperator does not fetch log entries for deferred jobs [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #55703:
URL: https://github.com/apache/airflow/pull/55703#issuecomment-3303828572
Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix(4561) - AWS BatchOperator does not fetch log entries for deferred jobs [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #55703:
URL: https://github.com/apache/airflow/pull/55703#discussion_r2352929375
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py:
##########
@@ -250,14 +250,22 @@ def execute(self, context: Context) -> str | None:
return self.job_id
- def execute_complete(self, context: Context, event: dict[str, Any] | None = None) -> str:
+ def execute_complete(
+ self, context: Context, event: dict[str, Any] | None = None
+ ) -> str:
validated_event = validate_execute_complete_event(event)
if validated_event["status"] != "success":
raise AirflowException(f"Error while running job: {validated_event}")
- self.log.info("Job completed.")
- return validated_event["job_id"]
+ self.job_id = validated_event["job_id"]
+
+ # Fetch logs if awslogs_enabled
+ if self.awslogs_enabled:
+ self.monitor_job(context) # fetch logs, no need to return
Review Comment:
We know the job is done by this point. Is `self.monitor_job` going to log anything about starting to monitor again for a second time that will confuse the users?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org