You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/05/03 16:24:33 UTC
[GitHub] [airflow] monti-python opened a new issue #15636: WasbHook does not delete blobs with slash characters in prefix mode
monti-python opened a new issue #15636:
URL: https://github.com/apache/airflow/issues/15636
**Apache Airflow version**: 2.0.1
**Environment**:
- **Cloud provider or hardware configuration**: docker container
- **OS** (e.g. from /etc/os-release): `Debian GNU/Linux 10 (buster)`
- **Kernel** (e.g. `uname -a`): `Linux 69a18af222ff 3.10.0-1160.15.2.el7.x86_64 #1 SMP Thu Jan 21 16:15:07 EST 2021 x86_64 GNU/Linux`
- **Install tools**: `pip`
- **Others**: `azure` extras
**What happened**:
`airflow.providers.microsoft.azure.hooks.wasb.WasbHook.delete_file` unable to delete blobs if both conditions below are true:
* `is_prefix` argument set to `True`
* the target blobs contain at least one '/' character in their names
**What you expected to happen**:
All files starting with the specified prefix are removed.
The problem is caused by this line: https://github.com/apache/airflow/blob/73187871703bce22783a42db3d3cec9045ee1de2/airflow/providers/microsoft/azure/hooks/wasb.py#L410
By not specifying `delimiter = ''` the listed blobs won't be terminal blobs if the target blobs happen to contain the default delimiter (`/`) in their name
**How to reproduce it**:
For example, consider the following scenario. We have a blob container `cnt` with two files:
* `parent/file1`
* `parent/file2`
The following call should delete both files:
``` python
wasb_hook= WasbHook(...)
wasb_hook.delete_file(
container_name='cnt',
blob_name='parent/',
is_prefix=True,
)
```
But instead we get an error `Blob(s) not found: ('parent/',)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] monti-python edited a comment on issue #15636: WasbHook does not delete blobs with slash characters in prefix mode
Posted by GitBox <gi...@apache.org>.
monti-python edited a comment on issue #15636:
URL: https://github.com/apache/airflow/issues/15636#issuecomment-831398880
I reopen it cause even specifying `check_options = {'delimiter': ''}` the method still breaks when calling `delete_blobs`; `delimiter` is not accepted as kwarg here: https://github.com/apache/airflow/blob/73187871703bce22783a42db3d3cec9045ee1de2/airflow/providers/microsoft/azure/hooks/wasb.py#L383
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #15636: WasbHook does not delete blobs with slash characters in prefix mode
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #15636:
URL: https://github.com/apache/airflow/issues/15636#issuecomment-831374843
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy closed issue #15636: WasbHook does not delete blobs with slash characters in prefix mode
Posted by GitBox <gi...@apache.org>.
ephraimbuddy closed issue #15636:
URL: https://github.com/apache/airflow/issues/15636
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] monti-python edited a comment on issue #15636: WasbHook does not delete blobs with slash characters in prefix mode
Posted by GitBox <gi...@apache.org>.
monti-python edited a comment on issue #15636:
URL: https://github.com/apache/airflow/issues/15636#issuecomment-831384476
It's possible to override default `delimiter` by using `check_options`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] monti-python closed issue #15636: WasbHook does not delete blobs with slash characters in prefix mode
Posted by GitBox <gi...@apache.org>.
monti-python closed issue #15636:
URL: https://github.com/apache/airflow/issues/15636
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] monti-python commented on issue #15636: WasbHook does not delete blobs with slash characters in prefix mode
Posted by GitBox <gi...@apache.org>.
monti-python commented on issue #15636:
URL: https://github.com/apache/airflow/issues/15636#issuecomment-831377567
Made a PR to fix it: #15637
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] monti-python commented on issue #15636: WasbHook does not delete blobs with slash characters in prefix mode
Posted by GitBox <gi...@apache.org>.
monti-python commented on issue #15636:
URL: https://github.com/apache/airflow/issues/15636#issuecomment-831398880
I reopen it cause it feels a bit hacky having to specify `check_options` just to remove all files based on a given prefix. I think it's worth considering the change in #15637
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] monti-python commented on issue #15636: WasbHook does not delete blobs with slash characters in prefix mode
Posted by GitBox <gi...@apache.org>.
monti-python commented on issue #15636:
URL: https://github.com/apache/airflow/issues/15636#issuecomment-831384476
Closing: It's possible to override default `delimiter` by using `check_options`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org