[GitHub] [airflow] potiuk commented on a diff in pull request #24019: Add celery broker configuration example using Amazon SQS

[GitBox <gi...@apache.org>]

potiuk commented on code in PR #24019:
URL: https://github.com/apache/airflow/pull/24019#discussion_r914833032


##########
docs/apache-airflow/executor/celery.rst:
##########
@@ -26,6 +26,17 @@ to work, you need to setup a Celery backend (**RabbitMQ**, **Redis**, ...) and
 change your ``airflow.cfg`` to point the executor parameter to
 ``CeleryExecutor`` and provide the related Celery settings.
 
+If you use Amazon SQS as a Celery broker, set environment variables like this:

Review Comment:
   Can you please add a note that this is experimental support/use at own risk. It would be great also to add something along "there are anecdotal evidences that Amazon SQS" might be used as the broker backend. 
   We do not run any tests for SQS as celery backend (as opposed to rabbitmq and redis - we have integration tests for both). 
   
   We do not want to give the users false impression that they can expect it to work and ask as question "I followed the documentation and it does not work" - unless we have CI/tests covering the scenario we are talking about. 
   
   Or maybe @ferruzzi @o-nikolas @john-jac - there could be some blog post or documentation from Amazon team to describe it. I believe MWAA uses SQS as broker and maybe one of you (or someone from the amazon developer relations team) coudl write a blog post on "How to use SQS as celery broker for Airflow? " that woudl be nice - and then any questions about it could come to Amazon- I am afraid until we make system tests/integration with Amazon, making a statement that "SQS works and here is the configuration" in Airlfow docs is a little too much "leap of faith" :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org