You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/05/27 03:54:02 UTC
[GitHub] [airflow] feluelle opened a new issue #9009: Use Terraform for System Testing
feluelle opened a new issue #9009:
URL: https://github.com/apache/airflow/issues/9009
**Description**
As @ashb @potiuk and me already shortly discussed in a slack thread..
We could use terraform for testing Airflow integrations with real services from providers.
Key benefits:
* Write declarative configuration files
* Plan and predict changes
* Create reproducible infrastructure
I didn't think about the implementation details (pytest + terraform) yet, but I think it would make it a lot easier using terraform instead of having to write a lot of system helper classes in python.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #9009: Use Terraform for System Testing
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #9009:
URL: https://github.com/apache/airflow/issues/9009#issuecomment-636726142
Fully Agree with Tomek. Especially with some setup/teardown that is quick and can be repeated for each test every time, using terraform makes no sense. But when you have longer setup that is more complex and takes quite some time, this might be definitely worthwhile.
The usual scenario for that will be
a) create the environment -> terraform script
b) iterate on the tests
alternating with
c) potentially changing target state of the environment - updating and re-running the terraform script applying the changes.
d) tear-down the environment (using terraform script to delete the environment).
This already pretty much reflects our complex "helpers" for system tests - basically when we really need a helper, seems that terraform scripts might to better job.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #9009: Use Terraform for System Testing
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #9009:
URL: https://github.com/apache/airflow/issues/9009#issuecomment-636726142
Fully Agree with Tomek. Especially with some setup/teardown that is quick and can be repeated for each test every time, using terraform makes no sense. But when you have longer setup that is more complex and takes quite some time, this might be definitely worthwhile.
The usual scenario for that will be
a) create the environment -> terraform script
b) iterate on the tests
alternating with
c) potentially changing target state of the environment - updating and re-running the terraform script applying the changes.
d) tear-down the environment (using terraform script to delete the environment).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #9009: Use Terraform for System Testing
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #9009:
URL: https://github.com/apache/airflow/issues/9009#issuecomment-639348034
Yeah. Airflow sources are best. But for that, you will need to create a HOST_AIRFLOW_SOURCES variable (similarly as I do with HOST_HOME) as an environment variable and mount it as `/opt/airlfow`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] szha edited a comment on issue #9009: Use Terraform for System Testing
Posted by GitBox <gi...@apache.org>.
szha edited a comment on issue #9009:
URL: https://github.com/apache/airflow/issues/9009#issuecomment-636414596
HashiCorp recently added a clause to their [term of evaluation](https://www.hashicorp.com/terms-of-evaluation) that forbids the usage of their enterprise software in China. While this may or may not affect this proposal, it does pose potential legal risk to contributors in China.
Update:
HashiCorp revised its terms to only include the enterprise version of vault.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #9009: Use Terraform for System Testing
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #9009:
URL: https://github.com/apache/airflow/issues/9009#issuecomment-638355905
> So what I have in mind is that the `def setUpClass` automatically `terraform init` and `apply`s the infra located in `tests/providers/*/infrastructure` for a file named as the example dag file and should end with `.tf` **if exists**
I like this idea!
> > Is creating a single GCS bucket worth adding terraform?
>
> @turbaszek in my opinion we should do it consistenly for all example_dags. Do you think it would be slow or just superfluous? Or what is your concern?
My main concern is that normally I would never use terraform to create single bucket. But it's not a strong opinion. I'm in favor of unified and opinionated approach 👌
> I think the biggest plus is that the infra automatically gets destroyed so we do not accidently let something exists longer as the test needs it.
Theoretically `tearDown` should always be executed so currently all infrastructure should also be destroyed. However, the advantage of using `.tf` is that everything we defined will be created and destroyed, so no place for discrepancies between setUp and tearDown.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on issue #9009: Use Terraform for System Testing
Posted by GitBox <gi...@apache.org>.
feluelle commented on issue #9009:
URL: https://github.com/apache/airflow/issues/9009#issuecomment-642485557
Terraform was introduced during #8877
For further integration i.e. automatisation of system tests we can also use terraform but that should be a separate issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #9009: Use Terraform for System Testing
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #9009:
URL: https://github.com/apache/airflow/issues/9009#issuecomment-639348512
And BTW. IT would be great to also add this mapping -v ${HOST_ARIFLOW_SOURCES}:/opt/airflow also to the other commands. We might wan to upload stuff to gcloud/aws as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle edited a comment on issue #9009: Use Terraform for System Testing
Posted by GitBox <gi...@apache.org>.
feluelle edited a comment on issue #9009:
URL: https://github.com/apache/airflow/issues/9009#issuecomment-638289406
So what I have in mind is that the `def setUpClass` automatically `terraform init` and `apply`s the infra located in `tests/providers/*/infrastructure` for a file named as the example dag file and should end with `.tf` **if exists**
What we could also do would be only running the terraform init for`tests/providers/*/infrastructure` once for all example dags when the providers system flag is used `--system amazon`.
> Is creating a single GCS bucket worth adding terraform?
@turbaszek in my opinion we should do it consistenly for all example_dags. Do you think it would be slow? Or what is your concern?
Under the hood terraform is also using the cli's we use to create resources for aws and google, right? So why not do it safely and more manageable using terraform? I think the biggest plus is that the infra automatically gets destroyed so we do not accidently let something exists longer as the test needs it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #9009: Use Terraform for System Testing
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #9009:
URL: https://github.com/apache/airflow/issues/9009#issuecomment-633894740
Absolutely!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on issue #9009: Use Terraform for System Testing
Posted by GitBox <gi...@apache.org>.
feluelle commented on issue #9009:
URL: https://github.com/apache/airflow/issues/9009#issuecomment-639345100
I have a question: When I am adding terraform to airflow and I want it through docker like it was done [here](https://github.com/apache/airflow/commit/a39e9a352050f96dedadc39ab3d985065971c98c) what volumes should I mount? I need to mount the .aws, .azure, etc. folder but also the folders containing the .tf files. How should I do it? Mount the whole airflow dir?
WDYT? @potiuk @turbaszek :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle edited a comment on issue #9009: Use Terraform for System Testing
Posted by GitBox <gi...@apache.org>.
feluelle edited a comment on issue #9009:
URL: https://github.com/apache/airflow/issues/9009#issuecomment-638289406
So what I have in mind is that the `def setUpClass` automatically `terraform init` and `apply`s the infra located in `tests/providers/*/infrastructure` for a file named as the example dag file and should end with `.tf` **if exists**
What we could also do would be only running the terraform init for`tests/providers/*/infrastructure` once for all example dags when the providers system flag is used `--system amazon` for example.
> Is creating a single GCS bucket worth adding terraform?
@turbaszek in my opinion we should do it consistenly for all example_dags. Do you think it would be slow or just superfluous? Or what is your concern?
Under the hood terraform is also using the cli's we use to create resources for aws and google, right? So why not do it safely and more manageable using terraform? I think the biggest plus is that the infra automatically gets destroyed so we do not accidently let something exists longer as the test needs it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #9009: Use Terraform for System Testing
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #9009:
URL: https://github.com/apache/airflow/issues/9009#issuecomment-633894740
Absolutley!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on issue #9009: Use Terraform for System Testing
Posted by GitBox <gi...@apache.org>.
feluelle commented on issue #9009:
URL: https://github.com/apache/airflow/issues/9009#issuecomment-638289406
So what I have in mind is that the `def setUpClass` automatically `terraform init`s the infra located in `tests/providers/*/infrastructure` for a file named as the example dag file and should end with `.tf` **if exists**
What we could also do would be only running the terraform init for`tests/providers/*/infrastructure` once for all example dags when the providers system flag is used `--system amazon`.
> Is creating a single GCS bucket worth adding terraform?
@turbaszek in my opinion we should do it consistenly for all example_dags. Do you think it would be slow? Or what is your concern?
Under the hood terraform is also using the cli's we use to create resources for aws and google, right? So why not do it safely and more manageable using terraform? I think the biggest plus is that the infra automatically gets destroyed so we do not accidently let something exists longer as the test needs it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] szha commented on issue #9009: Use Terraform for System Testing
Posted by GitBox <gi...@apache.org>.
szha commented on issue #9009:
URL: https://github.com/apache/airflow/issues/9009#issuecomment-636414596
HashiCorp recently added a clause to their [term of evaluation](https://www.hashicorp.com/terms-of-evaluation) that forbids the usage of their enterprise software in China. While this may or may not affect this proposal, it does pose potential legal risk to contributors in China.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle edited a comment on issue #9009: Use Terraform for System Testing
Posted by GitBox <gi...@apache.org>.
feluelle edited a comment on issue #9009:
URL: https://github.com/apache/airflow/issues/9009#issuecomment-638289406
So what I have in mind is that the `def setUpClass` automatically `terraform init` and `apply`s the infra located in `tests/providers/*/infrastructure` for a file named as the example dag file and should end with `.tf` **if exists**
What we could also do would be only running the terraform init for`tests/providers/*/infrastructure` once for all example dags when the providers system flag is used `--system amazon` for example.
> Is creating a single GCS bucket worth adding terraform?
@turbaszek in my opinion we should do it consistenly for all example_dags. Do you think it would be slow? Or what is your concern?
Under the hood terraform is also using the cli's we use to create resources for aws and google, right? So why not do it safely and more manageable using terraform? I think the biggest plus is that the infra automatically gets destroyed so we do not accidently let something exists longer as the test needs it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #9009: Use Terraform for System Testing
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #9009:
URL: https://github.com/apache/airflow/issues/9009#issuecomment-636710304
> I think it would make it a lot easier using terraform instead of having to write a lot of system helper classes in python.
In my opinion it depends... Is creating a single GCS bucket worth adding terraform? However, I fully support the idea for usting terraform to setup complex environment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #9009: Use Terraform for System Testing
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #9009:
URL: https://github.com/apache/airflow/issues/9009#issuecomment-633648606
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle closed issue #9009: Use Terraform for System Testing
Posted by GitBox <gi...@apache.org>.
feluelle closed issue #9009:
URL: https://github.com/apache/airflow/issues/9009
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org