You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/04/22 06:45:07 UTC

[GitHub] [airflow] Yao-ATG opened a new issue, #23162: GCSToGCSOperator ignores replace parameter when there is no wildcard

Yao-ATG opened a new issue, #23162:
URL: https://github.com/apache/airflow/issues/23162

   ### Apache Airflow Provider(s)
   
   google
   
   ### Versions of Apache Airflow Providers
   
   Latest
   
   ### Apache Airflow version
   
   2.2.5 (latest released)
   
   ### Operating System
   
   MacOS 12.2.1
   
   ### Deployment
   
   Composer
   
   ### Deployment details
   
   _No response_
   
   ### What happened
   
   Ran the same DAG twice with 'replace = False', in the second run files are overwritten anyway.
   source_object does not include wildcard.
   
   Not sure whether this incorrect behavior happens to "with wildcard" scenario, but from source code
   https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/transfers/gcs_to_gcs.py 
   in line 346 (inside _copy_source_with_wildcard) we have
         if not self.replace:
   but in _copy_source_without_wildcard we don't check self.replace at all.
   
   ### What you think should happen instead
   
   When 'replace = False', the second run should skip copying files since they are already there.
   
   ### How to reproduce
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] sahutd commented on issue #23162: GCSToGCSOperator ignores replace parameter when there is no wildcard

Posted by GitBox <gi...@apache.org>.
sahutd commented on issue #23162:
URL: https://github.com/apache/airflow/issues/23162#issuecomment-1111429679

   @potiuk i am interested in filing a fix for this[i might be facing a similar issue described here]
   
   as i have not developed airflow in the past, it might take me some time. in the meantime, if someone wants to file a fix, go for it :) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #23162: GCSToGCSOperator ignores replace parameter when there is no wildcard

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23162:
URL: https://github.com/apache/airflow/issues/23162#issuecomment-1109031455

   Can you add a PR to fix it please? That's the best way to make it happen quickly otherwise it will have to wait for somene to pick it up @Yao-ATG 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] spatocode commented on issue #23162: GCSToGCSOperator ignores replace parameter when there is no wildcard

Posted by GitBox <gi...@apache.org>.
spatocode commented on issue #23162:
URL: https://github.com/apache/airflow/issues/23162#issuecomment-1112080803

   > @potiuk i am interested in filing a fix for this[i might be facing a similar issue described here]
   > 
   > as i have not developed airflow in the past, it might take me some time. in the meantime, if someone wants to file a fix, go for it :)
   
   I'll be picking this up, then.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #23162: GCSToGCSOperator ignores replace parameter when there is no wildcard

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #23162: GCSToGCSOperator ignores replace parameter when there is no wildcard
URL: https://github.com/apache/airflow/issues/23162


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] spatocode commented on issue #23162: GCSToGCSOperator ignores replace parameter when there is no wildcard

Posted by GitBox <gi...@apache.org>.
spatocode commented on issue #23162:
URL: https://github.com/apache/airflow/issues/23162#issuecomment-1117330856

   @potiuk I raised a [PR](https://github.com/apache/airflow/pull/23340) to fix this. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org