[GitHub] [airflow] yegeniy commented on issue #6281: Run batches of (self-terminating) EMR JobFlows [AIRFLOW-5763]

[GitBox <gi...@apache.org>]

yegeniy commented on issue #6281: Run batches of (self-terminating) EMR JobFlows [AIRFLOW-5763]
URL: https://github.com/apache/airflow/pull/6281#issuecomment-548026133
 
 
   > couldn't this be implemented as multiple EmrCreateJobFlowOperators and
   some sensors at the end
   
   Yes, one of the use-cases I described could be replaced with existing EMR
   operators and sensors. I probably shouldn’t have emphasized this use-case
   as it’s not the main/novel use case.
   
   The novel use-case is that this operator allows for dynamic behavior at
   DagRun time without overwriting the DAG definition itself (that is, we get
   to keep the history in the UI and have parallel runs each with different
   definitions).
   
   I agree that this isn’t a common use case for Airflow, but internally I’ve
   found it to be a handy way to quickly iterate, develop, and run AWS
   EMR-based workflows that require multiple clusters to execute in sequence.
   
   Please feel free to close out this PR if it doesn’t belong in
   `apache/airflow`. I can just leave the code up on my branch if someone
   finds it useful in the future.
   
   On Wed, Oct 30, 2019 at 8:09 AM Ash Berlin-Taylor <no...@github.com>
   wrote:
   
   > I'm not sure this operator is really useful enough to belong in Airflow
   > core. If I'm reading the comments/docs correctly couldn't this be
   > implemented as multiple EmrCreateJobFlowOperators and some sensors at the
   > end?
   >
   > Right now I don't feel this needs a whole new operator to support it.
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/airflow/pull/6281?email_source=notifications&email_token=AACHPQUVAKYUETQ45EHKTOTQRF2NZA5CNFSM4I6MYH4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECT5OUA#issuecomment-547870544>,
   > or unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/AACHPQW7DPGYNPW2T4WAFM3QRF2NZANCNFSM4I6MYH4A>
   > .
   >
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services