You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/05/17 11:11:33 UTC
[GitHub] [airflow] randr97 opened a new issue #8890: Add AWS S3 create and delete bucket operator
randr97 opened a new issue #8890:
URL: https://github.com/apache/airflow/issues/8890
**Description**
AWS missing operators include CREATE and DELETE S3 operator. This issue caters to creating these operators. Also writing system tests for the same.
**Use case / motivation**
Many organisations would want to utilise such operators using xcoms for their projects. Especially when S3 is used with redshift and some ETL process has to be deployed to pull data from and dump data to a data lake.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] randr97 edited a comment on issue #8890: [WIP] Add AWS S3 create and delete bucket operator
Posted by GitBox <gi...@apache.org>.
randr97 edited a comment on issue #8890:
URL: https://github.com/apache/airflow/issues/8890#issuecomment-629805198
> @randr97 is this to create and delete buckets or objects in buckets? If it is to create/delete buckets, how would you interact with Terraform state backends?
Hi @prateeksan, this is to create and delete S3 buckets.
I did not get as to why we would need to interact with Terraform state backends.
It would be help if you could explain the use case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] randr97 closed issue #8890: Add AWS S3 create and delete bucket operator
Posted by GitBox <gi...@apache.org>.
randr97 closed issue #8890:
URL: https://github.com/apache/airflow/issues/8890
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] prateeksan commented on issue #8890: [WIP] Add AWS S3 create and delete bucket operator
Posted by GitBox <gi...@apache.org>.
prateeksan commented on issue #8890:
URL: https://github.com/apache/airflow/issues/8890#issuecomment-629800789
@randr97 is this to create and delete buckets or objects in buckets? If it is to create/delete buckets, how would you interact with Terraform state backends?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] prateeksan commented on issue #8890: [WIP] Add AWS S3 create and delete bucket operator
Posted by GitBox <gi...@apache.org>.
prateeksan commented on issue #8890:
URL: https://github.com/apache/airflow/issues/8890#issuecomment-629808178
@randr97 a lot of organizations manage infrastructure like S3 buckets through infrastructure-as-code scripts, usually Terraform since that is what AWS supports natively. The way Terraform works is that it maintains the state of all infrastructure in a backend (usually remote). Let's say someone configured an s3 bucket and deployed it using Terraform, if your DAG then deletes that bucket, it will create a mismatch between the Terraform state and the reality of the deployment (TF state will presume the bucket exists when in fact it doesn't).
I still think this is a useful feature, but I am wondering if there is a way for the operator to accept a flag that somehow communicates to a TF backend in case a bucket is deleted.
Also keep in mind that your create bucket operator should have all the required configuration options for IAM roles, ACL permissions etc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] randr97 commented on issue #8890: [WIP] Add AWS S3 create and delete bucket operator
Posted by GitBox <gi...@apache.org>.
randr97 commented on issue #8890:
URL: https://github.com/apache/airflow/issues/8890#issuecomment-629808815
> @randr97 a lot of organizations manage infrastructure like S3 buckets through infrastructure-as-code scripts, usually Terraform since that is what AWS supports natively. The way Terraform works is that it maintains the state of all infrastructure in a backend (usually remote). Let's say someone configured an s3 bucket and deployed it using Terraform, if your DAG then deletes that bucket, it will create a mismatch between the Terraform state and the reality of the deployment (TF state will presume the bucket exists when in fact it doesn't).
>
> I still think this is a useful feature, but I am wondering if there is a way for the operator to accept a flag that somehow communicates to a TF backend in case a bucket is deleted.
>
> Also keep in mind that your create bucket operator should have all the required configuration options for IAM roles, ACL permissions etc.
Oh now I get the issue. So there should be a mechanism where airflow could talk to TF backend so that for eg if some keys have to be created via terraform and dag deletes the bucket it should technically fail!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] randr97 commented on issue #8890: [WIP] Add AWS S3 create and delete bucket operator
Posted by GitBox <gi...@apache.org>.
randr97 commented on issue #8890:
URL: https://github.com/apache/airflow/issues/8890#issuecomment-629805198
> @randr97 is this to create and delete buckets or objects in buckets? If it is to create/delete buckets, how would you interact with Terraform state backends?
Hi @prateeksan, this is to create and delete S3 buckets.
I did not get as to why we would need to interact with Terraform state backends.
It would be help if you could explain the use case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] randr97 edited a comment on issue #8890: [WIP] Add AWS S3 create and delete bucket operator
Posted by GitBox <gi...@apache.org>.
randr97 edited a comment on issue #8890:
URL: https://github.com/apache/airflow/issues/8890#issuecomment-629808815
> @randr97 a lot of organizations manage infrastructure like S3 buckets through infrastructure-as-code scripts, usually Terraform since that is what AWS supports natively. The way Terraform works is that it maintains the state of all infrastructure in a backend (usually remote). Let's say someone configured an s3 bucket and deployed it using Terraform, if your DAG then deletes that bucket, it will create a mismatch between the Terraform state and the reality of the deployment (TF state will presume the bucket exists when in fact it doesn't).
>
> I still think this is a useful feature, but I am wondering if there is a way for the operator to accept a flag that somehow communicates to a TF backend in case a bucket is deleted.
>
> Also keep in mind that your create bucket operator should have all the required configuration options for IAM roles, ACL permissions etc.
Oh now I get the issue. So there should be a mechanism where airflow could talk to TF backend so that for eg if some keys have to be created via terraform and dag deletes the bucket it should technically fail!
Would be great if we could discuss this in detail on slack??
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] prateeksan commented on issue #8890: [WIP] Add AWS S3 create and delete bucket operator
Posted by GitBox <gi...@apache.org>.
prateeksan commented on issue #8890:
URL: https://github.com/apache/airflow/issues/8890#issuecomment-629811872
Something like that, I'll message you on Slack.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org