[GitHub] [airflow] jaketf commented on pull request #9593: Improve handling Dataproc cluster creation with ERROR state

[GitBox <gi...@apache.org>]

jaketf commented on pull request #9593:
URL: https://github.com/apache/airflow/pull/9593#issuecomment-668844747


   > BTW. Offtop - but should not we think about adding a Terraform/Terragrunt operator to Airflow ? I'd say it might be a good idea to have such an operator with some pre-defined ways on how to get terraform/terragrunt scripts in and how to integrate with airflow's JINJA templating.
   
   Having two levels of rendering (JINJA / hcl string interpolation) sounds like a great way to have hard to debug situations of "What level of this rendering is going wrong". And what really would by dynamic between task runs from an infra perspective?
   
   I think a terraform hook might be a nice feature but would take some careful design. I vaguely remember this being brought up on slack or dev list but can't seem to find it. I found myself wanting it to bring up / tear down a CI composer environment during sleeping hours as cost cutting measure.
   
   For OSS terraform / terragrunt this could be really tricky:
   1. It means the execution environment must have terraform / terragrunt binaries (plus download remote modules referenced or source for local modules). would we provide an airflow extra for this? would this be up to the user? The latter seems sad and not easy to do with Composer specifically (could be easier if you build your own airflow images to toss terraform in and run on kubernetes).
   2. you need an easy way need to sync terraform source from a repo (unless you imagine single resource type things that might be embedded in DAG code).
   I think it would mostly end up as a Kubernetes Pod Operator that happens to run terraform.
   
   However for terraform enterprise hook might be much simpler (as the the execution environment and source syncing  and state management become not airflow's problem).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org