You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/30 18:22:03 UTC
[GitHub] [airflow] mancunian1792 opened a new issue #22628: Enhancement: SQSSensor to stop polling after some pre-defined tries.
mancunian1792 opened a new issue #22628:
URL: https://github.com/apache/airflow/issues/22628
### Description
**Problem**: There can be days where this queue doesn't get any message. When that happens, my SQS Sensor continues to poll till it receives a message. Since i want it to make as real-time as possible, this prevents me from choosing a logical schedule interval for the DAG.
**Question**: How would you implement for cases where there may be a mix of lots of messages in a day and no message at all? People who post messages may work in short bursts (difficult to predict when they'd do that.). What's the ideal scenario for anyone to use an SQS sensor?
### Use case/motivation
**My Use case**: Pick up a message from SQS and run a set of tasks. (Lots of data transformations and integrations happen here that it makes sense to use airflow and not dump everything in a script.)
**Suggestions**: If there is a max_try parameter to the sensor my whole issue would be solved. Poll 'x' (parameterized. default can be infinite till you get a message.) number of times and if you don't receive a message by then, then write an empty array to xcom for key messages. This way, one can control the run time the dag/this particular task and hence control the schedule interval to work to their needs. Although, changes the very definition of polling but for airflow's case, a max_try parameter kinda makes sense to me
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #22628: Enhancement: SQSSensor to stop polling after some pre-defined tries.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #22628:
URL: https://github.com/apache/airflow/issues/22628#issuecomment-1084882028
But you can do it today: `Max_try = Timeout/ poke_interval`. Just a matter of setting the right values. Converting it into a discussion in case there are more things to discuss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #22628: Enhancement: SQSSensor to stop polling after some pre-defined tries.
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #22628:
URL: https://github.com/apache/airflow/issues/22628#issuecomment-1083468399
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org