You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "jack (JIRA)" <ji...@apache.org> on 2018/08/29 07:18:00 UTC

[jira] [Created] (AIRFLOW-2975) Add support for Amazon cloudwatch (computing power at will)

jack created AIRFLOW-2975:
-----------------------------

             Summary: Add support for Amazon cloudwatch (computing power at will)
                 Key: AIRFLOW-2975
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2975
             Project: Apache Airflow
          Issue Type: New Feature
    Affects Versions: 1.10.0
            Reporter: jack
             Fix For: 2.0.0, 1.10.1


Some of have one machine that runs airfow…

While we can scale up the executor to have many resources over different servers this is consider to be expensive.

There is another solution.. using Amazon Cloudwatch:

[https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/]

 

This enable the user to create and close EC2 machines on specific intervals for specific tasks.

Basically if I have 50 DAGS to run on 1PM-3PM and few dags to run on other hours there is no point in paying for 2nd server 24/7. 

This could be an enhancement to one of the executors that know to work with more than one server.

 

 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)