You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/13 10:19:26 UTC
[GitHub] [airflow] aslantar opened a new issue #16967: GCStoGCS operator not working when replace=False and source_object != destination_object
aslantar opened a new issue #16967:
URL: https://github.com/apache/airflow/issues/16967
The GCStoGCS operator using the _replace=False_ flag and specifying _source_object != destination_object_, will not work properly, because it is comparing [apples](https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/transfers/gcs_to_gcs.py#L351) with [pears](https://github.com/apache/airflow/blob/0dd70de9492444d52e0afaadd7a510fc8a95369c/airflow/providers/google/cloud/transfers/gcs_to_gcs.py#L357).
Cause: the _prefix_ in the _pears_ case needs to be calculated from the _destination_object_, however, it is coming from the _source_object_.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] aslantar commented on issue #16967: GCStoGCS operator not working when replace=False and source_object != destination_object
Posted by GitBox <gi...@apache.org>.
aslantar commented on issue #16967:
URL: https://github.com/apache/airflow/issues/16967#issuecomment-878999248
Possible solution:
```
existing_objects = hook.list(self.destination_bucket, prefix=prefix_, delimiter=delimiter)
# add an additional case
if self.destination_object is not None:
existing_objects = existing_objects.replace(prefix_, self.destination_object, 1)
objects = set(objects) - set(existing_objects)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] aslantar commented on issue #16967: GCStoGCS operator not working when replace=False and source_object != destination_object
Posted by GitBox <gi...@apache.org>.
aslantar commented on issue #16967:
URL: https://github.com/apache/airflow/issues/16967#issuecomment-879076674
Yes. I never contributed, but I'll find out and let you know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #16967: GCStoGCS operator not working when replace=False and source_object != destination_object
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #16967:
URL: https://github.com/apache/airflow/issues/16967#issuecomment-879055711
Would you like to provide a PR with a fix @aslantar ? happy to review it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] aslantar closed issue #16967: GCStoGCS operator not working when replace=False and source_object != destination_object
Posted by GitBox <gi...@apache.org>.
aslantar closed issue #16967:
URL: https://github.com/apache/airflow/issues/16967
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16967: GCStoGCS operator not working when replace=False and source_object != destination_object
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16967:
URL: https://github.com/apache/airflow/issues/16967#issuecomment-879055711
Would you like to provide a PR with a fix @aslantar - happy to review it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] aslantar commented on issue #16967: GCStoGCS operator not working when replace=False and source_object != destination_object
Posted by GitBox <gi...@apache.org>.
aslantar commented on issue #16967:
URL: https://github.com/apache/airflow/issues/16967#issuecomment-880782346
@potiuk From my side it is ready for review #16991. I'd especially be interested how I can improve wrt. your coding guidelines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #16967: GCStoGCS operator not working when replace=False and source_object != destination_object
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #16967:
URL: https://github.com/apache/airflow/issues/16967#issuecomment-878965338
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org