You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "isaiahiruoha (via GitHub)" <gi...@apache.org> on 2025/07/31 18:28:10 UTC
[PR] Fix AWS Lambda executor error handling for DLQ vs task failures [airflow]
isaiahiruoha opened a new pull request, #53990:
URL: https://github.com/apache/airflow/pull/53990
Enhanced error handling in the AWS Lambda executor to better distinguish between Lambda service failures (DLQ) and task execution failures.
Added check to properly identify DLQ failures and updated error messages to clearly indicate whether the failure was at the Lambda service level (timeout/crash/memory) or during task execution.
Previously, logging failures from the DLQ contained the incorrect field `return_code` instead of `command` which is located in the DLQ message body.
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix AWS Lambda executor error handling for DLQ vs task failures [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #53990:
URL: https://github.com/apache/airflow/pull/53990#discussion_r2246436331
##########
providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/lambda_executor.py:
##########
@@ -398,8 +398,8 @@ def process_queue(self, queue_url: str):
self.log.warning("Deleting the message to avoid processing it again.")
self.sqs_client.delete_message(QueueUrl=queue_url, ReceiptHandle=receipt_handle)
continue
- return_code = body.get("return_code")
ser_task_key = body.get("task_key")
+ return_code = body.get("return_code")
Review Comment:
Small nit: but these don't really need to be swapped right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix AWS Lambda executor error handling for DLQ vs task failures [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas merged PR #53990:
URL: https://github.com/apache/airflow/pull/53990
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix AWS Lambda executor error handling for DLQ vs task failures [airflow]
Posted by "isaiahiruoha (via GitHub)" <gi...@apache.org>.
isaiahiruoha commented on code in PR #53990:
URL: https://github.com/apache/airflow/pull/53990#discussion_r2246189998
##########
providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/lambda_executor.py:
##########
@@ -425,17 +426,27 @@ def process_queue(self, queue_url: str):
"Successful Lambda invocation for task %s received from SQS queue.", task_key
)
else:
- # In this case the Lambda likely started but failed at run time since we got a non-zero
- # return code. We could consider retrying these tasks within the executor, because this _likely_
- # means the Airflow task did not run to completion, however we can't be sure (maybe the
- # lambda runtime code has a bug and is returning a non-zero when it actually passed?). So
- # perhaps not retrying is the safest option.
self.fail(task_key)
- self.log.error(
- "Lambda invocation for task: %s has failed to run with return code %s",
- task_key,
- return_code,
- )
+ if queue_url == self.dlq_url and return_code == None:
+ # DLQ failure: AWS Lambda service could not complete the invocation after retries.
+ # This indicates a Lambda-level failure (timeout, memory limit, crash, etc.)
+ # where the function was unable to successfully execute to return a result.
+ self.log.error(
+ "Lambda invocation for task: %s failed at the service level (from DLQ). Command: %s",
Review Comment:
I agree, I went back and looked at some DLQ messages and the command field does not seem to be very helpful, ill adjust the logging to include the additional information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix AWS Lambda executor error handling for DLQ vs task failures [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #53990:
URL: https://github.com/apache/airflow/pull/53990#discussion_r2246178113
##########
providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/lambda_executor.py:
##########
@@ -425,17 +426,27 @@ def process_queue(self, queue_url: str):
"Successful Lambda invocation for task %s received from SQS queue.", task_key
)
else:
- # In this case the Lambda likely started but failed at run time since we got a non-zero
- # return code. We could consider retrying these tasks within the executor, because this _likely_
- # means the Airflow task did not run to completion, however we can't be sure (maybe the
- # lambda runtime code has a bug and is returning a non-zero when it actually passed?). So
- # perhaps not retrying is the safest option.
self.fail(task_key)
- self.log.error(
- "Lambda invocation for task: %s has failed to run with return code %s",
- task_key,
- return_code,
- )
+ if queue_url == self.dlq_url and return_code == None:
+ # DLQ failure: AWS Lambda service could not complete the invocation after retries.
+ # This indicates a Lambda-level failure (timeout, memory limit, crash, etc.)
+ # where the function was unable to successfully execute to return a result.
+ self.log.error(
+ "Lambda invocation for task: %s failed at the service level (from DLQ). Command: %s",
Review Comment:
The command is pretty large in the AF3 case, so I'm not sure we should log it (it doesn't really have helpful information either). I also think you have some very helpful stuff in the comment that might be nice to include in the error message itself to give the user some context and pointers?
What about:
```suggestion
"DLQ message received: Lambda invocation for task: %s was unable to successfully execute. This likely indicates a Lambda-level failure (timeout, memory limit, crash, etc.).",
```
##########
providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/lambda_executor.py:
##########
@@ -425,17 +426,27 @@ def process_queue(self, queue_url: str):
"Successful Lambda invocation for task %s received from SQS queue.", task_key
)
else:
- # In this case the Lambda likely started but failed at run time since we got a non-zero
- # return code. We could consider retrying these tasks within the executor, because this _likely_
- # means the Airflow task did not run to completion, however we can't be sure (maybe the
- # lambda runtime code has a bug and is returning a non-zero when it actually passed?). So
- # perhaps not retrying is the safest option.
self.fail(task_key)
- self.log.error(
- "Lambda invocation for task: %s has failed to run with return code %s",
- task_key,
- return_code,
- )
+ if queue_url == self.dlq_url and return_code == None:
+ # DLQ failure: AWS Lambda service could not complete the invocation after retries.
+ # This indicates a Lambda-level failure (timeout, memory limit, crash, etc.)
+ # where the function was unable to successfully execute to return a result.
+ self.log.error(
+ "Lambda invocation for task: %s failed at the service level (from DLQ). Command: %s",
+ task_key,
+ command,
+ )
+ else:
+ # In this case the Lambda likely started but failed at run time since we got a non-zero
+ # return code. We could consider retrying these tasks within the executor, because this _likely_
+ # means the Airflow task did not run to completion, however we can't be sure (maybe the
+ # lambda runtime code has a bug and is returning a non-zero when it actually passed?). So
+ # perhaps not retrying is the safest option.
+ self.log.error(
+ "Lambda invocation for task: %s has failed to run with return code %s",
Review Comment:
We should also probably differentiate this one a bit more as well.
```suggestion
"Lambda invocation for task: %s completed but the underlying Airflow task has returned a non-zero exit code %s",
```
I'm also wondering if this branch should be log.debug? Since it's very common for airlfow tasks to fail for a myriad of reasons, and we don't necessarily need a error log in the scheduler logs for each one of those. Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix AWS Lambda executor error handling for DLQ vs task failures [airflow]
Posted by "isaiahiruoha (via GitHub)" <gi...@apache.org>.
isaiahiruoha commented on code in PR #53990:
URL: https://github.com/apache/airflow/pull/53990#discussion_r2246452050
##########
providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/lambda_executor.py:
##########
@@ -398,8 +398,8 @@ def process_queue(self, queue_url: str):
self.log.warning("Deleting the message to avoid processing it again.")
self.sqs_client.delete_message(QueueUrl=queue_url, ReceiptHandle=receipt_handle)
continue
- return_code = body.get("return_code")
ser_task_key = body.get("task_key")
+ return_code = body.get("return_code")
Review Comment:
True, I had originally switched the order when adding the `command` extraction since `ser_task_key` was always needed, and then `return_code` and `command` were based on queue type. But with no `command` in the output, switching does not make sense. Adjusted, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix AWS Lambda executor error handling for DLQ vs task failures [airflow]
Posted by "isaiahiruoha (via GitHub)" <gi...@apache.org>.
isaiahiruoha commented on code in PR #53990:
URL: https://github.com/apache/airflow/pull/53990#discussion_r2246330252
##########
providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/lambda_executor.py:
##########
@@ -425,17 +426,27 @@ def process_queue(self, queue_url: str):
"Successful Lambda invocation for task %s received from SQS queue.", task_key
)
else:
- # In this case the Lambda likely started but failed at run time since we got a non-zero
- # return code. We could consider retrying these tasks within the executor, because this _likely_
- # means the Airflow task did not run to completion, however we can't be sure (maybe the
- # lambda runtime code has a bug and is returning a non-zero when it actually passed?). So
- # perhaps not retrying is the safest option.
self.fail(task_key)
- self.log.error(
- "Lambda invocation for task: %s has failed to run with return code %s",
- task_key,
- return_code,
- )
+ if queue_url == self.dlq_url and return_code == None:
+ # DLQ failure: AWS Lambda service could not complete the invocation after retries.
+ # This indicates a Lambda-level failure (timeout, memory limit, crash, etc.)
+ # where the function was unable to successfully execute to return a result.
+ self.log.error(
+ "Lambda invocation for task: %s failed at the service level (from DLQ). Command: %s",
+ task_key,
+ command,
+ )
+ else:
+ # In this case the Lambda likely started but failed at run time since we got a non-zero
+ # return code. We could consider retrying these tasks within the executor, because this _likely_
+ # means the Airflow task did not run to completion, however we can't be sure (maybe the
+ # lambda runtime code has a bug and is returning a non-zero when it actually passed?). So
+ # perhaps not retrying is the safest option.
+ self.log.error(
+ "Lambda invocation for task: %s has failed to run with return code %s",
Review Comment:
I agree, I've made the adjustment in the latest commit, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix AWS Lambda executor error handling for DLQ vs task failures [airflow]
Posted by "isaiahiruoha (via GitHub)" <gi...@apache.org>.
isaiahiruoha commented on code in PR #53990:
URL: https://github.com/apache/airflow/pull/53990#discussion_r2246214111
##########
providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/lambda_executor.py:
##########
@@ -425,17 +426,27 @@ def process_queue(self, queue_url: str):
"Successful Lambda invocation for task %s received from SQS queue.", task_key
)
else:
- # In this case the Lambda likely started but failed at run time since we got a non-zero
- # return code. We could consider retrying these tasks within the executor, because this _likely_
- # means the Airflow task did not run to completion, however we can't be sure (maybe the
- # lambda runtime code has a bug and is returning a non-zero when it actually passed?). So
- # perhaps not retrying is the safest option.
self.fail(task_key)
- self.log.error(
- "Lambda invocation for task: %s has failed to run with return code %s",
- task_key,
- return_code,
- )
+ if queue_url == self.dlq_url and return_code == None:
+ # DLQ failure: AWS Lambda service could not complete the invocation after retries.
+ # This indicates a Lambda-level failure (timeout, memory limit, crash, etc.)
+ # where the function was unable to successfully execute to return a result.
+ self.log.error(
+ "Lambda invocation for task: %s failed at the service level (from DLQ). Command: %s",
+ task_key,
+ command,
+ )
+ else:
+ # In this case the Lambda likely started but failed at run time since we got a non-zero
+ # return code. We could consider retrying these tasks within the executor, because this _likely_
+ # means the Airflow task did not run to completion, however we can't be sure (maybe the
+ # lambda runtime code has a bug and is returning a non-zero when it actually passed?). So
+ # perhaps not retrying is the safest option.
+ self.log.error(
+ "Lambda invocation for task: %s has failed to run with return code %s",
Review Comment:
But I suppose since it is a catchall for a large number of possible failures, log.debug would make sense in nearly all situations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix AWS Lambda executor error handling for DLQ vs task failures [airflow]
Posted by "isaiahiruoha (via GitHub)" <gi...@apache.org>.
isaiahiruoha commented on code in PR #53990:
URL: https://github.com/apache/airflow/pull/53990#discussion_r2246210169
##########
providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/lambda_executor.py:
##########
@@ -425,17 +426,27 @@ def process_queue(self, queue_url: str):
"Successful Lambda invocation for task %s received from SQS queue.", task_key
)
else:
- # In this case the Lambda likely started but failed at run time since we got a non-zero
- # return code. We could consider retrying these tasks within the executor, because this _likely_
- # means the Airflow task did not run to completion, however we can't be sure (maybe the
- # lambda runtime code has a bug and is returning a non-zero when it actually passed?). So
- # perhaps not retrying is the safest option.
self.fail(task_key)
- self.log.error(
- "Lambda invocation for task: %s has failed to run with return code %s",
- task_key,
- return_code,
- )
+ if queue_url == self.dlq_url and return_code == None:
+ # DLQ failure: AWS Lambda service could not complete the invocation after retries.
+ # This indicates a Lambda-level failure (timeout, memory limit, crash, etc.)
+ # where the function was unable to successfully execute to return a result.
+ self.log.error(
+ "Lambda invocation for task: %s failed at the service level (from DLQ). Command: %s",
+ task_key,
+ command,
+ )
+ else:
+ # In this case the Lambda likely started but failed at run time since we got a non-zero
+ # return code. We could consider retrying these tasks within the executor, because this _likely_
+ # means the Airflow task did not run to completion, however we can't be sure (maybe the
+ # lambda runtime code has a bug and is returning a non-zero when it actually passed?). So
+ # perhaps not retrying is the safest option.
+ self.log.error(
+ "Lambda invocation for task: %s has failed to run with return code %s",
Review Comment:
I definitely see the argument because from the scheduler's perspective it has completed its job and task based failures will have more meaningful logging within their respective logs.
My only concern would be in situations similar to what I experienced where a task that began as an attempted lambda timeout to cause a catastrophic failure, but was later retried and provided a return code, falling into the second branch and avoiding the DLQ.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix AWS Lambda executor error handling for DLQ vs task failures [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #53990:
URL: https://github.com/apache/airflow/pull/53990#discussion_r2246275473
##########
providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/lambda_executor.py:
##########
@@ -425,17 +426,27 @@ def process_queue(self, queue_url: str):
"Successful Lambda invocation for task %s received from SQS queue.", task_key
)
else:
- # In this case the Lambda likely started but failed at run time since we got a non-zero
- # return code. We could consider retrying these tasks within the executor, because this _likely_
- # means the Airflow task did not run to completion, however we can't be sure (maybe the
- # lambda runtime code has a bug and is returning a non-zero when it actually passed?). So
- # perhaps not retrying is the safest option.
self.fail(task_key)
- self.log.error(
- "Lambda invocation for task: %s has failed to run with return code %s",
- task_key,
- return_code,
- )
+ if queue_url == self.dlq_url and return_code == None:
+ # DLQ failure: AWS Lambda service could not complete the invocation after retries.
+ # This indicates a Lambda-level failure (timeout, memory limit, crash, etc.)
+ # where the function was unable to successfully execute to return a result.
+ self.log.error(
+ "Lambda invocation for task: %s failed at the service level (from DLQ). Command: %s",
+ task_key,
+ command,
+ )
+ else:
+ # In this case the Lambda likely started but failed at run time since we got a non-zero
+ # return code. We could consider retrying these tasks within the executor, because this _likely_
+ # means the Airflow task did not run to completion, however we can't be sure (maybe the
+ # lambda runtime code has a bug and is returning a non-zero when it actually passed?). So
+ # perhaps not retrying is the safest option.
+ self.log.error(
+ "Lambda invocation for task: %s has failed to run with return code %s",
Review Comment:
Yeah, that's fair, but in that situation the task still got marked as failed, and that will show up in the Airflow UI, scheduler logs, etc. So users will be more than aware it failed and can investigate and figure out the cause. I'm just not sure the Lambda executor _also_ needs to raise alarms for those failures. Feels redundant and like it most often will just cause excess churn in the logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org