You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "pabrrs (via GitHub)" <gi...@apache.org> on 2023/03/09 16:10:08 UTC

[GitHub] [airflow] pabrrs opened a new issue, #30000: Add PURGE flag to S3ToSnowflakeOperator

pabrrs opened a new issue, #30000:
URL: https://github.com/apache/airflow/issues/30000

   ### Description
   
   Snowflake `COPY` operator supports `PURGE=[TRUE|FALSE]` to delete files in S3 after they're copied into the tables. It would be nice to replicate this behavior in `S3ToSnowflakeOperator` with a flag.
   
   [See snowflake COPY options docs](https://docs.snowflake.com/en/sql-reference/sql/copy-into-table#copy-options-copyoptions)
   
   ### Use case/motivation
   
   In my use case, the files doesn't need to keep In the bucket after being inserted into tables.
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] Taragolis commented on issue #30000: Add PURGE flag to S3ToSnowflakeOperator

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #30000:
URL: https://github.com/apache/airflow/issues/30000#issuecomment-1462646662

   And as I could see [CopyFromExternalStageToSnowflakeOperator](https://airflow.apache.org/docs/apache-airflow-providers-snowflake/stable/_api/airflow/providers/snowflake/transfers/copy_into_snowflake/index.html#airflow.providers.snowflake.transfers.copy_into_snowflake.CopyFromExternalStageToSnowflakeOperator) supports `copy_options` argument


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #30000: Add PURGE flag to S3ToSnowflakeOperator

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #30000:
URL: https://github.com/apache/airflow/issues/30000#issuecomment-1462334541

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] pabrrs closed issue #30000: Add PURGE flag to S3ToSnowflakeOperator

Posted by "pabrrs (via GitHub)" <gi...@apache.org>.
pabrrs closed issue #30000: Add PURGE flag to S3ToSnowflakeOperator
URL: https://github.com/apache/airflow/issues/30000


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] Taragolis commented on issue #30000: Add PURGE flag to S3ToSnowflakeOperator

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #30000:
URL: https://github.com/apache/airflow/issues/30000#issuecomment-1462639841

   Nice! Just one thing `S3ToSnowflakeOperator` is deprecated and changes should be addressed to [`CopyFromExternalStageToSnowflakeOperator`](https://github.com/apache/airflow/blob/0fd4fc7f35f0e12bef2c9615acf9651e9f2cec72/airflow/providers/snowflake/transfers/copy_into_snowflake.py#L30)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] pabrrs commented on issue #30000: Add PURGE flag to S3ToSnowflakeOperator

Posted by "pabrrs (via GitHub)" <gi...@apache.org>.
pabrrs commented on issue #30000:
URL: https://github.com/apache/airflow/issues/30000#issuecomment-1462654858

   @Taragolis thanks for referencing `CopyFromExternalStageToSnowflakeOperator`, this works fine for me. 
   Since `S3ToSnowflakeOperator` is deprecated, I believe not worth to implement it. 
   
   Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #30000: Add PURGE flag to S3ToSnowflakeOperator

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #30000:
URL: https://github.com/apache/airflow/issues/30000#issuecomment-1462628860

   WHOA. I was literally 2 PRs too late to get the 30000 one ! congrats @pabrrs !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] josh-fell commented on issue #30000: Add PURGE flag to S3ToSnowflakeOperator

Posted by "josh-fell (via GitHub)" <gi...@apache.org>.
josh-fell commented on issue #30000:
URL: https://github.com/apache/airflow/issues/30000#issuecomment-1462497738

   Nice! @pabrrs All yours!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org