You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "cliebBS (via GitHub)" <gi...@apache.org> on 2023/11/28 18:45:48 UTC
[I] Unable to filter SQS messages using jsonpath [airflow]
cliebBS opened a new issue, #35932:
URL: https://github.com/apache/airflow/issues/35932
### Apache Airflow version
2.7.3
### What happened
When using the `SqsSensor` with `message_filtering="jsonpath"` and both `message_filtering_config` and `message_filtering_match_values` set, the following error is produced:
```
[2023-11-28, 18:24:44 UTC] {taskinstance.py:1937} ERROR - [findings_sync manual__2023-11-27T16:12:36+00:00 notify_started_test_receive 18] Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/amazon/aws/sensors/sqs.py", line 144, in execute
super().execute(context=context)
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/sensors/base.py", line 257, in execute
raise e
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/sensors/base.py", line 239, in execute
poke_return = self.poke(context)
^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/amazon/aws/sensors/sqs.py", line 187, in poke
messages = process_response(
^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/amazon/aws/utils/sqs.py", line 49, in process_response
messages = filter_messages(
^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/amazon/aws/utils/sqs.py", line 63, in filter_messages
return filter_messages_jsonpath(messages, message_filtering_match_values, message_filtering_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/amazon/aws/utils/sqs.py", line 87, in filter_messages_jsonpath
or any(result.value in message_filtering_match_values for result in results)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/amazon/aws/utils/sqs.py", line 87, in <genexpr>
or any(result.value in message_filtering_match_values for result in results)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: unhashable type: 'dict'
```
### What you think should happen instead
The messages should be filtered using the provided JsonPath query and the provided match values.
### How to reproduce
Sorry, all I have available is a FIFO queue, and the Amazon providers for both SQS and SNS are unable to send messages to FIFO queues due to their lack of support for the `message_deduplication_id` field, which FIFO queues require to be set.
### Operating System
MacOS 14.1.1 and Amazon Linux 2
### Versions of Apache Airflow Providers
8.10.0
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to filter SQS messages using jsonpath [airflow]
Posted by "cliebBS (via GitHub)" <gi...@apache.org>.
cliebBS commented on issue #35932:
URL: https://github.com/apache/airflow/issues/35932#issuecomment-1830480183
Sorry, not sure how to make a PR automatically attach to GitHub tickets, so here's my fix: https://github.com/apache/airflow/pull/35933
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to filter SQS messages using jsonpath [airflow]
Posted by "cliebBS (via GitHub)" <gi...@apache.org>.
cliebBS commented on issue #35932:
URL: https://github.com/apache/airflow/issues/35932#issuecomment-1832045361
Turns out this isn't an issue. While I was writing a test for my fix, I discovered that I was actually matching against a different part of the message than I thought I was, meaning that the returned value was a dict instead of a string like I was expecting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to filter SQS messages using jsonpath [airflow]
Posted by "cliebBS (via GitHub)" <gi...@apache.org>.
cliebBS closed issue #35932: Unable to filter SQS messages using jsonpath
URL: https://github.com/apache/airflow/issues/35932
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to filter SQS messages using jsonpath [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #35932:
URL: https://github.com/apache/airflow/issues/35932#issuecomment-1830492017
> Sorry, not sure how to make a PR automatically attach to GitHub tickets, so here's my fix: #35933
No need to raise issue you can open PR directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org