You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/08 08:12:07 UTC
[GitHub] [airflow] baolsen opened a new issue #16880: Improve AWS SQS Sensor
baolsen opened a new issue #16880:
URL: https://github.com/apache/airflow/issues/16880
**Description**
Improve the AWS SQS Sensor as follows:
+ Check the HTTP status code in AWS response and raise Exception if not 200 - best practice
+ Add optional visibility_timeout parameter
+ Add a customisable / overrideable filter capability so we can filter/ignore irrelevant messages
**Use case / motivation**
I'd like to make the SQS sensor more flexible to enable the following use case:
+ A single queue can be used as a channel for messages from multiple event sources and or multiple targets
+ We need a way to filter and ignore messages not relevant to us, which other processes are looking for
**Are you willing to submit a PR?**
Yes, please assign to me
**Related Issues**
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] baolsen commented on issue #16880: Improve AWS SQS Sensor
Posted by GitBox <gi...@apache.org>.
baolsen commented on issue #16880:
URL: https://github.com/apache/airflow/issues/16880#issuecomment-876931178
That is awesome, thank you for sharing. I'll scratch that one from this request :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #16880: Improve AWS SQS Sensor
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #16880:
URL: https://github.com/apache/airflow/issues/16880
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] baolsen commented on issue #16880: Improve AWS SQS Sensor
Posted by GitBox <gi...@apache.org>.
baolsen commented on issue #16880:
URL: https://github.com/apache/airflow/issues/16880#issuecomment-876333454
Edit: The "clearest reference" link doesn't even mention checking the HTTP code which is definitely in the responses I have.... So yea not clear at all then ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] baolsen commented on issue #16880: Improve AWS SQS Sensor
Posted by GitBox <gi...@apache.org>.
baolsen commented on issue #16880:
URL: https://github.com/apache/airflow/issues/16880#issuecomment-876332322
> > Check the HTTP status code in AWS response and raise Exception if not 200 - best practice
>
> Doesn't Boto do that for us already?
Yea I have a gripe about the usability/accuracy of the AWS documentation sometimes.
Some of the boto3 APIs do indeed provide a managed capability so we don't need to check return codes.
But many (most?) others do not, especially older ones. It's not always clear in the documentation which is which... and there are no "since" tags on the APIs either.
I have noticed that if an API is managed it is usually called out as such, but even then some managed APIs don't fully isolate the user from certain issues ;)
For the SQS receive_message API in particular -
The boto3 response does not document explicitly that it returns ResponseMetadata, but in practice it definitely does return it :)
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs.html#SQS.Client.receive_message
At the top of the boto3 SQS page, client section, additional information section, they do link to this:
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-api-responses.html
So you kind of have to stumble upon it... but that is the clearest reference I could find.
There is also this guide:
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html
Which refers to "Common errors" but is not very clear on where to detect the errors.
This developer guide link also doesn't clarify, in fact it doesn't mention the response metadata at all:
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/working-with-messages.html#handling-request-errors
In my SQS responses I do see the ResponseMetadata element that can be (and presumably should be) checked.
So the check I would like to add is the usual:
```
if not response['ResponseMetadata']['HTTPStatusCode'] == 200:
raise Exception('An HTTP error occurred')
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #16880: Improve AWS SQS Sensor
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #16880:
URL: https://github.com/apache/airflow/issues/16880#issuecomment-876299980
> Check the HTTP status code in AWS response and raise Exception if not 200 - best practice
Doesn't Boto do that for us already?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #16880: Improve AWS SQS Sensor
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #16880:
URL: https://github.com/apache/airflow/issues/16880#issuecomment-876351905
https://github.com/boto/botocore/blob/23ee17f5446c78167ff442302471f9928c3b4b7c/docs/source/client_upgrades.rst#error-handling
Botocore handles it -- we don't have to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org