You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/25 15:10:40 UTC

[GitHub] [airflow] mik-laj opened a new issue #10550: Terraform provider based on Stable API

mik-laj opened a new issue #10550:
URL: https://github.com/apache/airflow/issues/10550

Hello,

If we have a [Stable API](https://airflow.readthedocs.io/en/latest/stable-rest-api-ref.html) ready, we can start working on tools that use it. I think we can start with the Terraform provider. Terraform. is a tool for building, changing, and versioning infrastructure as a code. A terraform provider is a way to integrate other services. Integrations already exist for a wide variety of services and software, including [Kubernetes](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs), [Google Cloud](https://www.terraform.io/docs/providers/google/index.html), [Keycloak](https://github.com/mrparkers/terraform-provider-keycloak).

@houqp started working on it and prepared [small POC](https://github.com/houqp/terraform-provider-airflow). Currently, this allows you to manage Airflow variables.

I discussed this topic with good citizen @ad-m and he said we can use [terraform-provider-openapi](https://github.com/dikhan/terraform-provider-openapi) to generate a terraform provider based on the OpenAPI specification. He pointed out that this is how the company for which he works developed the [terraform-provider-hyperone](https://github.com/hyperonecom/terraform-provider-hyperone). This is also an implementation idea that we can explore.

I talked to @houqp , He pointed out that implementations may be affected by the range of resources we want available. Not all resources in the API makes sense as resources in Terraform, eg TaskInstnace.

What do you think about the Terraform provider for Airflow? What use cases should we support? What are your ideas for implementing this concept?

Best regards,
Kamil Breguła

CC: @jaketf @potiuk

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ad-m commented on issue #10550: Terraform provider based on Stable API

Posted by GitBox <gi...@apache.org>.

ad-m commented on issue #10550:
URL: https://github.com/apache/airflow/issues/10550#issuecomment-682375114


   The use of "terraform-provider-openapi" allows for possible future Terraform updates automatically. It is only necessary to prepare the tests so that the library can be updated automatically. Tests - taking into account the existing standards - can be prepared on the basis of documentation that must be created.
   Such an update can happen when a new field is added (it's easy to miss Terraform updates then and Python developers do not need to have the appropriate skills to update), or a change to the Terraform API.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jaketf commented on issue #10550: Terraform provider based on Stable API

Posted by GitBox <gi...@apache.org>.

jaketf commented on issue #10550:
URL: https://github.com/apache/airflow/issues/10550#issuecomment-680150041

I could see use cases for the following (in decreasing order of how likely I'd be to recommend managing in terraform):
1. Pool: this feels rather infrastructure level could be something naturally in terraform's realm
1. Variable: add / update a references to infra provisioned by terraform that will be referenced by DAGs
1. Connection: add connection for db provisioned by terraform (though this seems worse than using secrets backend)
1. TriggerDagRun: this is more dubious but perhaps there is a scenario where after provisioning a bunch of infrastructure there is need to kick off an airflow DAG.

I think providing resources / data sources for other resources in the API would likely encourage pretty dangerous patterns leading to unpredictable terraform applies. Keep in mind that terraform likes to be authoritative about things so any values we expect to drift will be confusing to terraform and it will try to correct this drift.

Personally I'd be more interested in use cases for the "opposite" integration (airflow running terraform https://github.com/apache/airflow/issues/10454)

It is good to give users options as there are many different organization that may have different needs / opinions.

I've worked on [terraform-cdap-provider](https://github.com/GoogleCloudPlatform/terraform-provider-cdap#:~:text=Terraform%20CDAP%20Provider,official%20Google%20or%20Hashicorp%20product) in the past which we basically implemented by hand.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] houqp commented on issue #10550: Terraform provider based on Stable API

Posted by GitBox <gi...@apache.org>.

houqp commented on issue #10550:
URL: https://github.com/apache/airflow/issues/10550#issuecomment-682368672


   I am on the same page as @jaketf with regards to list of resources that makes sense to be managed by terraform. It's actually quite small that implementing them by hand incurs very little work.
   
   That said, i am interested in knowing other people's thought on this and see if there are valid use-cases to manage more resource types from Terraform. If we are to support many more resource types, then terraform-provider-openapi is probably the better route to go since we don't have that many resources working on a hand rolled version.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org