[GitHub] [airflow] aelzeiny edited a comment on issue #7030: [AIRFLOW-6440][AIP-29] Add AWS Fargate Executor

[GitBox <gi...@apache.org>]

aelzeiny edited a comment on issue #7030: [AIRFLOW-6440][AIP-29] Add AWS Fargate Executor
URL: https://github.com/apache/airflow/pull/7030#issuecomment-593982317
 
 
   So a bit of an update. Thank you all for your suggestions. Most recently I've been on the phone with the #sig-autoscaling slack channel to investigate whether or not this code will make it into the Airflow repo.
   
   Since AWS ECS/Fargate is a proprietary technology, it's hard to maintain the breeze environment and develop integration tests. There are very valid concerns about maintenance going forward if we lack an AWS ECS cluster to test on. For this reason, this executor is a hard-pass.
   
   It's also true that you can scale the Celery Executor on ECS or Fargate, and you would have a better time doing so. As an added bonus there would be 0 code involved. As of December, this is possible because Fargate added EFS support. I'm currently working on the cloud-formation stack that would spin this up. Effectively, The AWS Scheduler would put messages in an SQS which is monitored through CloudWatch which is hooked into an Application-AutoScaler for the AWS Service which triggers a Capacity Provider. AWS technically gives you the tools; it's just hard to string it all together.
   
   For now I'll make this Executor an optional plugin for Airflow. Thank you all for contributing and your suggestions.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services