You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/11/09 01:53:05 UTC
[GitHub] [airflow] ephraimbuddy opened a new pull request #12188: Upgrade azure blob and fileshare to v12
ephraimbuddy opened a new pull request #12188:
URL: https://github.com/apache/airflow/pull/12188
This PR attempts to upgrade azure storage blob and azure storage fileshare to version 12
- [ ] Upgrade azure storage blob
- [ ] Upgrade azure file share
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-759958121
Hey @ephraimbuddy - > I rebased your change on top of my latest "snowflake" fix and added one commit on top that should solve the dependency problem entirely (I hope at least :)
In my change I also again simplified a bit the dependency changes, no more EPOCH changes are needed and everything should work as expected- whenever you change setup.py/setup.cfg not onle dependencies are upgraded to latest possible, but also they are always reinstalled from scratch in this case - which means that the azure-storage dependency will not be installed in the first place. I hope we will finally squash it. I think after this one is merged we will also be able to instlall apache-beam as first-class citzien and finally get rid of all exceptions in dependency processing !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-767117077
Yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735307045
[The Workflow run](https://github.com/apache/airflow/actions/runs/389221470) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735214518
> > HI @potiuk , I'm not sure why the azure-blob-storage is not getting installed, despite that I added it to setup.py
>
> In this case you need "upgrade to latest dependencies" label and rebase it . I just added the label. please rebase to the the latest master :)
Shouldn't a change to setup.py/setup.cfg be all that is needed @potiuk ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-766078595
Constraints updated: https://github.com/apache/airflow/commit/cf7bae9fcb19a338cdd2a8a8d839fcdadde2c628:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-765733895
[The Workflow run](https://github.com/apache/airflow/actions/runs/504824467) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#discussion_r529423807
##########
File path: airflow/providers/microsoft/azure/hooks/wasb.py
##########
@@ -16,94 +16,147 @@
# specific language governing permissions and limitations
# under the License.
#
-"""
-This module contains integration with Azure Blob Storage.
+"""This module contains integration with Azure Blob Storage."""
-It communicate via the Window Azure Storage Blob protocol. Make sure that a
-Airflow connection of type `wasb` exists. Authorization can be done by supplying a
-login (=Storage account name) and password (=KEY), or login and SAS token in the extra
-field (see connection `wasb_default` for an example).
Review comment:
Ah.. mistake, I'll add it back
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-749821556
[The Workflow run](https://github.com/apache/airflow/actions/runs/439058510) is cancelling this PR. Building image for the PR has been cancelled
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735305099
At this commit https://github.com/apache/airflow/pull/12188/commits/7f9e52310fcd49f363aa4d88a3d2e3e1a3c096eb, it was installing well. I did some manual test which worked fine
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735221233
Please, @potiuk , help me to sort the dependencies out. For some reasons, I ended up having `azure-storage` and `azure-storage-blob` in the environment. I had to uninstall `azure-storage`, `azure-storage-blob` and `azure-storage-file` then re-install `azure-storage-blob` and `azure-storage-file` before I could have working `azure-storage-blob` and `azure-storage-file` installs.
I ran the system tests for `test_file_to_wasb_system.py` and `test_fileshare_system.py`. They passed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-734915636
Hey @ephraimbuddy - This is the last blocker for the #12636 , but for now I am going to add some conditional imports and make the relevant tssts as `xfail`, but once this on is ready to merge, I will remove those!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-760038953
I pushed it again aftter #13669
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-761868812
[The Workflow run](https://github.com/apache/airflow/actions/runs/492097869) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-753686824
[The Workflow run](https://github.com/apache/airflow/actions/runs/459851922) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735428640
> I found this from my email https://github.com/apache/airflow/actions/runs/365256644
But I believe that one worked :) (at least with regards to tests/deps)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#discussion_r528200000
##########
File path: airflow/providers/microsoft/azure/hooks/wasb.py
##########
@@ -163,11 +213,100 @@ def read_file(self, container_name: str, blob_name: str, **kwargs):
:type container_name: str
:param blob_name: Name of the blob.
:type blob_name: str
- :param kwargs: Optional keyword arguments that
- `BlockBlobService.create_blob_from_path()` takes.
+ :param kwargs: Optional keyword arguments that `BlobClient.download_blob` takes.
:type kwargs: object
"""
- return self.connection.get_blob_to_text(container_name, blob_name, **kwargs).content
+ return self.download(container_name, blob_name, **kwargs).readall()
+
+ def upload(
+ self,
+ container_name,
+ blob_name,
+ data,
+ blob_type: str = 'BlockBlob',
+ length: Optional[int] = None,
+ **kwargs,
+ ) -> Dict[str, Any]:
+ """
+ Creates a new blob from a data source with automatic chunking.
+
+ :param container_name: The name of the container to upload data
+ :type container_name: str
+ :param blob_name: The name of the blob to upload. This need not exist in the container
+ :type blob_name: str
+ :param data: The blob data to upload
+ :param blob_type: The type of the blob. This can be either BlockBlob,
+ PageBlob or AppendBlob. The default value is BlockBlob.
Review comment:
```suggestion
:param blob_type: The type of the blob. This can be either ``BlockBlob``,
``PageBlob`` or ``AppendBlob``. The default value is ``BlockBlob``.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#discussion_r528199853
##########
File path: airflow/providers/microsoft/azure/hooks/wasb.py
##########
@@ -16,94 +16,145 @@
# specific language governing permissions and limitations
# under the License.
#
-"""
-This module contains integration with Azure Blob Storage.
+"""This module contains integration with Azure Blob Storage."""
-It communicate via the Window Azure Storage Blob protocol. Make sure that a
-Airflow connection of type `wasb` exists. Authorization can be done by supplying a
-login (=Storage account name) and password (=KEY), or login and SAS token in the extra
-field (see connection `wasb_default` for an example).
-"""
-from azure.storage.blob import BlockBlobService
+from typing import Any, Dict, List, Optional
+
+from azure.core.exceptions import ResourceExistsError, ResourceNotFoundError
+from azure.identity import ClientSecretCredential
+from azure.storage.blob import BlobClient, BlobServiceClient, ContainerClient, StorageStreamDownloader
from airflow.exceptions import AirflowException
from airflow.hooks.base_hook import BaseHook
class WasbHook(BaseHook):
"""
- Interacts with Azure Blob Storage through the ``wasb://`` protocol.
-
- These parameters have to be passed in Airflow Data Base: account_name and account_key.
-
- Additional options passed in the 'extra' field of the connection will be
- passed to the `BlockBlockService()` constructor. For example, authenticate
- using a SAS token by adding {"sas_token": "YOUR_TOKEN"}.
+ Interacts with Azure Blob Storage.
:param wasb_conn_id: Reference to the wasb connection.
:type wasb_conn_id: str
+ :param public_read: Whether an anonymous public read access should be used. default is False
+ :type public_read: bool
"""
- def __init__(self, wasb_conn_id: str = 'wasb_default') -> None:
+ def __init__(self, wasb_conn_id: str = 'wasb_default', public_read: bool = False) -> None:
super().__init__()
self.conn_id = wasb_conn_id
+ self.public_read = public_read
self.connection = self.get_conn()
- def get_conn(self) -> BlockBlobService:
- """Return the BlockBlobService object."""
+ def get_conn(self) -> BlobServiceClient:
+ """Return the BlobServiceClient object."""
conn = self.get_connection(self.conn_id)
- service_options = conn.extra_dejson
- return BlockBlobService(account_name=conn.login, account_key=conn.password, **service_options)
+ extra = conn.extra_dejson
- def check_for_blob(self, container_name, blob_name, **kwargs):
+ if self.public_read:
+ # Here we use anonymous public read
+ # more info
+ # https://docs.microsoft.com/en-us/azure/storage/blobs/storage-manage-access-to-resources
+ return BlobServiceClient(account_url=conn.host)
+ if extra.get('connection_string'):
+ # connection_string auth takes priority
+ return BlobServiceClient.from_connection_string(extra.get('connection_string'))
+ if extra.get('shared_access_key'):
+ # using shared access key
+ return BlobServiceClient(account_url=conn.host, credential=extra.get('shared_access_key'))
+ if extra.get('tenant_id'):
+ # use Active Directory auth
+ app_id = conn.login
+ app_secret = conn.password
+ token_credential = ClientSecretCredential(extra.get('tenant_id'), app_id, app_secret)
+ return BlobServiceClient(account_url=conn.host, credential=token_credential)
+ if extra.get('sas_token'):
+ return BlobServiceClient(account_url=extra.get('sas_token'))
+ else:
+ raise AirflowException('Unknown connection type')
+
+ def _get_container_client(self, container_name: str) -> ContainerClient:
"""
- Check if a blob exists on Azure Blob Storage.
+ Instantiates a container client
+
+ :param container_name: The name of the container
+ :type container_name: str
+ :return: ContainerClient
+ """
+ return self.connection.get_container_client(container_name)
+
+ def _get_blob_client(self, container_name: str, blob_name: str) -> BlobClient:
+ """
+ Instantiates a blob client
+ :param container_name: The name of the blob container
+ :type container_name: str
+ :param blob_name: The name of the blob. This needs not be existing
+ :type blob_name: str
+ """
+ container_client = self.create_container(container_name)
+ return container_client.get_blob_client(blob_name)
+ def check_for_blob(self, container_name: str, blob_name: str, **kwargs) -> bool:
+ """
+ Check if a blob exists on Azure Blob Storage.
:param container_name: Name of the container.
:type container_name: str
:param blob_name: Name of the blob.
:type blob_name: str
- :param kwargs: Optional keyword arguments that
- `BlockBlobService.exists()` takes.
+ :param kwargs: Optional keyword arguments for `BlobClient.get_blob_properties` takes.
:type kwargs: object
:return: True if the blob exists, False otherwise.
:rtype: bool
"""
- return self.connection.exists(container_name, blob_name, **kwargs)
+ try:
+ self._get_blob_client(container_name, blob_name).get_blob_properties(**kwargs)
+ except ResourceNotFoundError:
+ return False
+ return True
- def check_for_prefix(self, container_name: str, prefix: str, **kwargs) -> bool:
+ def check_for_prefix(self, container_name: str, prefix: str, **kwargs):
"""
Check if a prefix exists on Azure Blob storage.
:param container_name: Name of the container.
:type container_name: str
:param prefix: Prefix of the blob.
:type prefix: str
- :param kwargs: Optional keyword arguments that
- `BlockBlobService.list_blobs()` takes.
+ :param kwargs: Optional keyword arguments that ContainerClient.walk_blobs takes
Review comment:
```suggestion
:param kwargs: Optional keyword arguments that ``ContainerClient.walk_blobs`` takes
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-759972938
> Hey @ephraimbuddy - > I rebased your change on top of my latest "snowflake" fix and added one commit on top that should solve the dependency problem entirely (I hope at least :)
>
> In my change I also again simplified a bit the dependency changes, no more EPOCH changes are needed and everything should work as expected- whenever you change setup.py/setup.cfg not onle dependencies are upgraded to latest possible, but also they are always reinstalled from scratch in this case - which means that the azure-storage dependency will not be installed in the first place. I hope we will finally squash it. I think after this one is merged we will also be able to instlall apache-beam as first-class citzien and finally get rid of all exceptions in dependency processing !
Awesome 😊. Not installing the azure-storage will solve it once
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735414184
[The Workflow run](https://github.com/apache/airflow/actions/runs/390078931) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#discussion_r528252804
##########
File path: docs/spelling_wordlist.txt
##########
@@ -1297,6 +1297,7 @@ uid
umask
un
unarchived
+uncommittedblobs
Review comment:
The test won't pass if this is removed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735216551
> Shouldn't a change to setup.py/setup.cfg be all that is needed @potiuk ?
Indeed. In this case it will not work. What @ephrain will need to do is to add || true after pip check (because the problem is that currently production image dependencies are still broken,
But in general case this label might be needed if you have some reason to force dependencies to latest ones.
This works as follows:
* In PRs we do not run "eager" upgrade strategy, be default we use "only-if-needed" strategy. This is in order to avoid new released dependencies breaking people's PR. PRs will use the "constrained" dependencies, unless the change in setup.py causes that those dependencies need to be upgraded because the "constrained" dependencies are outside of the requirements.
* We only use "eager" upgrade strategy in two cases:
1) when we run direct push or schedule from master
2) when "upgrade to latest dependencies" label is set on a PR.
The first case is needed to automatically upgrade constraints to the latest "good dependencies". If the dependencies (after eager upgrade) are non-conflicting (they should not be now) and if all tests passes. the constraint file in the right branch is automatically updated with those new constraints,
The second case is only needed really if you want to try to fix failing upgrade to latest dependencies. If the 'master" build will start failing when trying to run "eager" update, we have to have some way to add extra limitations and test it. This is exactly where we need to set the "upgrade to latest dependencies" label. Such PR will take the latest limitation from setup.py/setup.cfg and run "eager" update and run all tests. This way we can easily work on fixing failing "eager" upgrade.
I hope thise explanation makes it is clear.
I am also going to add some explanation in error messages in all those places where upgrade with eager dependencies might fail and provide instructions what to do in those cases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-731583308
@mik-laj @turbaszek @feluelle, can you please review, thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-762277590
[The Workflow run](https://github.com/apache/airflow/actions/runs/493650921) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#discussion_r529420707
##########
File path: airflow/providers/microsoft/azure/hooks/wasb.py
##########
@@ -16,94 +16,147 @@
# specific language governing permissions and limitations
# under the License.
#
-"""
-This module contains integration with Azure Blob Storage.
+"""This module contains integration with Azure Blob Storage."""
-It communicate via the Window Azure Storage Blob protocol. Make sure that a
-Airflow connection of type `wasb` exists. Authorization can be done by supplying a
-login (=Storage account name) and password (=KEY), or login and SAS token in the extra
-field (see connection `wasb_default` for an example).
Review comment:
This doc seems useful -- does it exist elsewhere?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735348159
> I tried this and it didn't work.
Can you please link to the build? I'd love to take a look.
> I saw that both `azure-storage-blob` and `azure-storage` are in https://raw.githubusercontent.com/apache/airflow/constraints-master/constraints-3.6.txt
Yep. they will be there until we refresh them. Constraints are not requirements, They do not determine what should be installed they merely tell "if you are installing this, it should be that version"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-734916470
> Hey @ephraimbuddy - This is the last blocker for the #12636 , but for now I am going to add some conditional imports and make the relevant tssts as `xfail`, but once this on is ready to merge, I will remove those!
Ok. I will work on it tonight
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-749738172
HEy @ephraimbuddy - can you rebase /fix static checks? We can take a look with @mik-laj and possibly merge it rather quickly to fix the other problems with the snowflake connector ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-765634319
Fixing it now
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735277002
BTW. I will likely add another label for that soon - using the mechanism we have, so that in the future it will be possible to simply add label "disable pip cache" and we should get this without all this hassle.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735276782
That should be rather easy I think. In this case we need to make a little more changes because we do not handle well "uninstalling of a previously installed packages from cache" but it can be done rather easily:
In the Dockerfile.ci please change those two lines - > change the AIRFLOW_CI_BUILD_EPOCH to 4, and AIRFLOW_PRE_CACHED_PIP_PACKAGES to "false" :
```
# By changing the CI build epoch we can force reinstalling Airflow from the current master
# It can also be overwritten manually by setting the AIRFLOW_CI_BUILD_EPOCH environment variable.
ARG AIRFLOW_CI_BUILD_EPOCH="3"
ENV AIRFLOW_CI_BUILD_EPOCH=${AIRFLOW_CI_BUILD_EPOCH}
ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="true"
ENV AIRFLOW_PRE_CACHED_PIP_PACKAGES=${AIRFLOW_PRE_CACHED_PIP_PACKAGES}
```
What those two changes mean:
* increasing the EPOCH causes invalidation of Docker Cache from this place. This means that for your PR the dockerfile will be rebuilt starting from that line.
* setting AIRFLOW_PRE_CACHED_PIP_PACKAGES to "false" disables the optimisation where we use the previously build images as cache for new builds - thanks to that we save a lot of time for image builds, but whatever was originally installed in the image remains there (including 'azure-storage' package).
If you change both values, your build will run longer, but it will not have the 'azure-storage' package installed at all (since you removed it from the setup.py).
When we see that all that works, We will just leave the EPOCH change. Once your change will be merge to master, all new builds will use the new master version of setup.py and will not install 'azure-storage' once the rebase to master and get your EPOCH change.
I hope it helps , ping me on slaack when you re-rerun it and still have some problems.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-732809672
> > Manually tested with this dag:
>
> Can you add an automatic test? It shouldn't be difficult, and it will allow us to test this integration more easily.
I'll add this test latest this weekend
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735428640
> I found this from my email https://github.com/apache/airflow/actions/runs/365256644
But I believe that one worked :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735208100
This is a new feature added last week.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-749738172
HEy @ephraimbuddy - can you rebase /fix static checks? We can take a look with @mik-laj and possibly merge it rather quickly to fix the other problems with the snowflake connection ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-759822470
> @ephraimbuddy Cancelled because of the long wait in the queue :( . I m afraid at least one more rebase will be needed :( sorry.
>
> On a good front #12881 upgrades snowflake and removes some limitation, so we are getting closer.
Nice
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735278941
Even when you think the problem is DNS, and you check it and find dns is not the problem, the problem is still always DNS
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb edited a comment on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ashb edited a comment on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735214518
> > HI @potiuk , I'm not sure why the azure-blob-storage is not getting installed, despite that I added it to setup.py
>
> In this case you need "upgrade to latest dependencies" label and rebase it . I just added the label. please rebase to the the latest master :)
Shouldn't a change to setup.py/setup.cfg be all that is needed @potiuk ?
Let me be clearer: why do we need this label? Setup.py changed. We should respect it automatically. Not doing so is a bug, not a feature.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #12188:
URL: https://github.com/apache/airflow/pull/12188
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-765817655
[The Workflow run](https://github.com/apache/airflow/actions/runs/505073217) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-767083620
This should be released in 2.0.1, right? @ephraimbuddy @potiuk
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-749921653
> When I ran breeze locally, I found that azure-storage 0.36 is still getting installed
Yes. We have extra protection to prevent image damage. In most cases, the constraints file is used to build the image, which contains a set of tested libraries. But we would like to update this file automatically sometimes, so we need the normal pip install to install the correct dependencies.
More info:
http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/installation.html
https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pinned-constraint-files
Recently, snowflake-connector-python dropped support for old azure library, so now snowflake and azure can't be installed, because pip will have trouble finding dependencies.
See: https://github.com/apache/airflow/pull/13195
In this PR, I am trying to improve compatibility with newer versions of Pinot and the pinot-python library, so I added the label "upgrade to newer dependencies" which means the image will be built without constraint, so the dependency installation may fail and it happened now. Several libraries have been updated and unfortunately, these dependencies do not work.
Here I have compiled a list of the differences between installing with and without a constraints file. Most are safe and will be updated soon in constraints.txt as well, but we have a few changes we will add to setup.py/setup.cfg, so users will be limited in using these libraries.
```
diff <(pip freeze | sort) <(curl https://raw.githubusercontent.com/apache/airflow/constraints-master/constraints-3.6.txt | sort)
< -e /opt/airflow
19c18
< JPype1==1.2.0
---
> JPype1==1.1.2
30c29
< Pygments==2.7.3
---
> Pygments==2.7.2
33,35c32,34
< SQLAlchemy==1.3.22
< Sphinx==3.4.0
< Unidecode==1.1.2
---
> SQLAlchemy==1.3.20
> Sphinx==3.3.1
> Unidecode==1.1.1
56c55
< aws-sam-translator==1.33.0
---
> aws-sam-translator==1.31.0
58c57
< azure-batch==10.0.0
---
> azure-batch==9.0.0
65c64
< azure-keyvault-keys==4.3.1
---
> azure-keyvault-keys==4.3.0
90c89
< cachetools==4.2.0
---
> cachetools==4.1.1
94c93
< certifi==2020.12.5
---
> certifi==2020.11.8
97c96
< cfn-lint==0.44.1
---
> cfn-lint==0.42.0
109c108
< coverage==5.3.1
---
> coverage==5.3
111c110
< cryptography==3.3.1
---
> cryptography==3.2.1
113,114c112,113
< cx-Oracle==8.1.0
< dask==2020.12.0
---
> cx-Oracle==8.0.1
> dask==2.30.0
133c132
< eventlet==0.30.0
---
> eventlet==0.29.1
135,136c134,135
< facebook-business==9.0.1
< fastavro==1.2.1
---
> facebook-business==9.0.0
> fastavro==1.2.0
142c141,142
< flower==0.9.7
---
> flask-swagger==0.2.13
> flower==0.9.5
144c144,145
< fsspec==0.8.5
---
> fsspec==0.8.4
> funcsigs==1.0.2
152c153
< google-api-core==1.24.1
---
> google-api-core==1.23.0
156c157
< google-auth==1.24.0
---
> google-auth==1.23.0
160,161c161,162
< google-cloud-bigquery==2.6.1
< google-cloud-bigtable==1.6.1
---
> google-cloud-bigquery==2.4.0
> google-cloud-bigtable==1.6.0
163c164
< google-cloud-core==1.5.0
---
> google-cloud-core==1.4.3
178c179
< google-cloud-storage==1.35.0
---
> google-cloud-storage==1.33.0
184,185c185,186
< google-crc32c==1.1.0
< google-resumable-media==1.2.0
---
> google-crc32c==1.0.0
> google-resumable-media==1.1.0
191c192
< grpcio==1.34.0
---
> grpcio==1.33.2
196,197c197,198
< humanize==3.2.0
< hvac==0.10.6
---
> humanize==3.1.0
> hvac==0.10.5
212c213
< isort==5.6.4
---
> isort==4.3.21
219,220c220,221
< jsonpatch==1.28
< jsonpickle==1.4.2
---
> jsonpatch==1.27
> jsonpickle==1.4.1
232c233
< libcst==0.3.16
---
> libcst==0.3.14
237c238
< marshmallow==3.10.0
---
> marshmallow==3.9.1
239c240
< mock==4.0.3
---
> mock==4.0.2
245,246c246,247
< msal==1.8.0
< msgpack==1.0.2
---
> msal==1.6.0
> msgpack==1.0.0
250c251
< multidict==5.1.0
---
> multidict==5.0.2
267c268
< packaging==20.8
---
> packaging==20.4
269c270
< pandas==1.1.5
---
> pandas==1.1.4
281c282
< pinotdb==0.3.3
---
> pinotdb==0.1.1
285c286
< pre-commit==2.9.3
---
> pre-commit==2.9.2
290c291
< proto-plus==1.13.0
---
> proto-plus==1.11.0
292c293
< psutil==5.8.0
---
> psutil==5.7.3
296,297c297,298
< py==1.10.0
< pyOpenSSL==20.0.1
---
> py==1.9.0
> pyOpenSSL==19.1.0
306,307c307,308
< pydruid==0.6.2
< pyenchant==3.2.0
---
> pydruid==0.6.1
> pyenchant==3.1.1
311,312c312,313
< pylint==2.6.0
< pymongo==3.11.2
---
> pylint==2.5.3
> pymongo==3.11.1
324,325c325,326
< pytest-xdist==2.2.0
< pytest==6.2.1
---
> pytest-xdist==2.1.0
> pytest==6.1.2
355c356
< sendgrid==6.4.8
---
> sendgrid==6.4.7
357,358c358,359
< sentry-sdk==0.19.5
< setproctitle==1.2.1
---
> sentry-sdk==0.19.4
> setproctitle==1.2
364c365
< snowflake-connector-python==2.3.7
---
> snowflake-connector-python==2.3.6
367c368
< soupsieve==2.1
---
> soupsieve==2.0.1
391c392
< tableauserverclient==0.14.1
---
> tableauserverclient==0.14.0
396c397
< testfixtures==6.17.0
---
> testfixtures==6.15.0
404c405
< tqdm==4.54.1
---
> tqdm==4.54.0
414c415
< vertica-python==1.0.1
---
> vertica-python==1.0.0
416c417
< virtualenv==20.2.2
---
> virtualenv==20.2.1
424c425
< yandexcloud==0.64.0
---
> yandexcloud==0.61.0
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735007445
Depends on #12673
When #12673 is merged, I will rebase and run the system tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-731584448
> Manually tested with this dag:
Can you add an automatic test? It shouldn't be difficult, and it will allow us to test this integration more easily.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#discussion_r528199794
##########
File path: airflow/providers/microsoft/azure/hooks/wasb.py
##########
@@ -16,94 +16,145 @@
# specific language governing permissions and limitations
# under the License.
#
-"""
-This module contains integration with Azure Blob Storage.
+"""This module contains integration with Azure Blob Storage."""
-It communicate via the Window Azure Storage Blob protocol. Make sure that a
-Airflow connection of type `wasb` exists. Authorization can be done by supplying a
-login (=Storage account name) and password (=KEY), or login and SAS token in the extra
-field (see connection `wasb_default` for an example).
-"""
-from azure.storage.blob import BlockBlobService
+from typing import Any, Dict, List, Optional
+
+from azure.core.exceptions import ResourceExistsError, ResourceNotFoundError
+from azure.identity import ClientSecretCredential
+from azure.storage.blob import BlobClient, BlobServiceClient, ContainerClient, StorageStreamDownloader
from airflow.exceptions import AirflowException
from airflow.hooks.base_hook import BaseHook
class WasbHook(BaseHook):
"""
- Interacts with Azure Blob Storage through the ``wasb://`` protocol.
-
- These parameters have to be passed in Airflow Data Base: account_name and account_key.
-
- Additional options passed in the 'extra' field of the connection will be
- passed to the `BlockBlockService()` constructor. For example, authenticate
- using a SAS token by adding {"sas_token": "YOUR_TOKEN"}.
+ Interacts with Azure Blob Storage.
:param wasb_conn_id: Reference to the wasb connection.
:type wasb_conn_id: str
+ :param public_read: Whether an anonymous public read access should be used. default is False
+ :type public_read: bool
"""
- def __init__(self, wasb_conn_id: str = 'wasb_default') -> None:
+ def __init__(self, wasb_conn_id: str = 'wasb_default', public_read: bool = False) -> None:
super().__init__()
self.conn_id = wasb_conn_id
+ self.public_read = public_read
self.connection = self.get_conn()
- def get_conn(self) -> BlockBlobService:
- """Return the BlockBlobService object."""
+ def get_conn(self) -> BlobServiceClient:
+ """Return the BlobServiceClient object."""
conn = self.get_connection(self.conn_id)
- service_options = conn.extra_dejson
- return BlockBlobService(account_name=conn.login, account_key=conn.password, **service_options)
+ extra = conn.extra_dejson
- def check_for_blob(self, container_name, blob_name, **kwargs):
+ if self.public_read:
+ # Here we use anonymous public read
+ # more info
+ # https://docs.microsoft.com/en-us/azure/storage/blobs/storage-manage-access-to-resources
+ return BlobServiceClient(account_url=conn.host)
+ if extra.get('connection_string'):
+ # connection_string auth takes priority
+ return BlobServiceClient.from_connection_string(extra.get('connection_string'))
+ if extra.get('shared_access_key'):
+ # using shared access key
+ return BlobServiceClient(account_url=conn.host, credential=extra.get('shared_access_key'))
+ if extra.get('tenant_id'):
+ # use Active Directory auth
+ app_id = conn.login
+ app_secret = conn.password
+ token_credential = ClientSecretCredential(extra.get('tenant_id'), app_id, app_secret)
+ return BlobServiceClient(account_url=conn.host, credential=token_credential)
+ if extra.get('sas_token'):
+ return BlobServiceClient(account_url=extra.get('sas_token'))
+ else:
+ raise AirflowException('Unknown connection type')
+
+ def _get_container_client(self, container_name: str) -> ContainerClient:
"""
- Check if a blob exists on Azure Blob Storage.
+ Instantiates a container client
+
+ :param container_name: The name of the container
+ :type container_name: str
+ :return: ContainerClient
+ """
+ return self.connection.get_container_client(container_name)
+
+ def _get_blob_client(self, container_name: str, blob_name: str) -> BlobClient:
+ """
+ Instantiates a blob client
+ :param container_name: The name of the blob container
Review comment:
```suggestion
:param container_name: The name of the blob container
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735240485
I read on this link https://pypi.org/project/azure-storage-file/ that `azure-storage` need to be uninstalled first before installing `azure-storage-file`. It seems to be the cause of this issue.
```
If you see azure-storage==0.36.0 (or any version below 0.36.0), uninstall it first
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-765984891
What a ride @ephraimbuddy @kaxil @mik-laj . I merged it now - I will now regenerate the constraints and remove the hack for azure-storage removal :).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735428805
> But I believe that one worked :)
Yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-749915051
When I ran breeze locally, I found that `azure-storage 0.36` is still getting installed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735348159
> I tried this and it didn't work.
Can you please link to the build? I'd love to take a look.
> I saw that both `azure-storage-blob` and `azure-storage` are in https://raw.githubusercontent.com/apache/airflow/constraints-master/constraints-3.6.txt
Yep. they will be there until we refresh them. Constraints are not requirements, They do not determine what shoudl be installed they merely tell "if you are installing this, it should be that version"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-765868868
Hey @ephraimbuddy -> I fixed the last failing static check with Breeze I hope.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-767117077
Yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-757805494
Hey @ephraimbuddy -> I think I finally figured out a good solution for the failures you had. It took some time but I think it's going to work. @mik-laj used it to upgrade dependencies on #13534 and ##13505 . So right now rebased should give consistent set of dependencies based on the updated setup.py:
:crossed_fingers:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-751835086
Hey @ephraimbuddy - would you mind rebasing it on tope of the latest master ? I have just merged a change where upgrades to newer dependencies should also work for prod image and I think with the latest changes in CI logs it will be much easier to find what could be wrong here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-752950877
@mik-laj fixed the pyjwt problems in master .... One more rebaase @ephraimbuddy :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-749921653
> When I ran breeze locally, I found that azure-storage 0.36 is still getting installed
Yes. We have extra protection to prevent image damage. In most cases, the constraints file is used to build the image, which contains a set of tested libraries. But we would like to update this file automatically sometimes, so we need the normal pip install to install the correct dependencies.
More info:
http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/installation.html
https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pinned-constraint-files
Recently, snowflake-connector-python dropped support for old azure library, so now snowflake and azure can't be installed, because pip will have trouble finding dependencies.
See: https://github.com/apache/airflow/pull/13195
In this PR, I am trying to improve compatibility with newer versions of Pinot and the pinot-python library, so I added the label "upgrade to newer dependencies" which means the image will be built without constraint, so the dependency installation may fail and it happened now. Several libraries have been updated and unfortunately, these dependencies do not work.
Here I have compiled a list of the differences between installing with and without a constraints file. Most are safe and will be updated soon in constraints.txt as well, but we have a few changes we will add to setup.py/setup.cfg, so users will be limited in using these libraries.
```diff
diff <(pip freeze | sort) <(curl https://raw.githubusercontent.com/apache/airflow/constraints-master/constraints-3.6.txt | sort)
< -e /opt/airflow
19c18
< JPype1==1.2.0
---
> JPype1==1.1.2
30c29
< Pygments==2.7.3
---
> Pygments==2.7.2
33,35c32,34
< SQLAlchemy==1.3.22
< Sphinx==3.4.0
< Unidecode==1.1.2
---
> SQLAlchemy==1.3.20
> Sphinx==3.3.1
> Unidecode==1.1.1
56c55
< aws-sam-translator==1.33.0
---
> aws-sam-translator==1.31.0
58c57
< azure-batch==10.0.0
---
> azure-batch==9.0.0
65c64
< azure-keyvault-keys==4.3.1
---
> azure-keyvault-keys==4.3.0
90c89
< cachetools==4.2.0
---
> cachetools==4.1.1
94c93
< certifi==2020.12.5
---
> certifi==2020.11.8
97c96
< cfn-lint==0.44.1
---
> cfn-lint==0.42.0
109c108
< coverage==5.3.1
---
> coverage==5.3
111c110
< cryptography==3.3.1
---
> cryptography==3.2.1
113,114c112,113
< cx-Oracle==8.1.0
< dask==2020.12.0
---
> cx-Oracle==8.0.1
> dask==2.30.0
133c132
< eventlet==0.30.0
---
> eventlet==0.29.1
135,136c134,135
< facebook-business==9.0.1
< fastavro==1.2.1
---
> facebook-business==9.0.0
> fastavro==1.2.0
142c141,142
< flower==0.9.7
---
> flask-swagger==0.2.13
> flower==0.9.5
144c144,145
< fsspec==0.8.5
---
> fsspec==0.8.4
> funcsigs==1.0.2
152c153
< google-api-core==1.24.1
---
> google-api-core==1.23.0
156c157
< google-auth==1.24.0
---
> google-auth==1.23.0
160,161c161,162
< google-cloud-bigquery==2.6.1
< google-cloud-bigtable==1.6.1
---
> google-cloud-bigquery==2.4.0
> google-cloud-bigtable==1.6.0
163c164
< google-cloud-core==1.5.0
---
> google-cloud-core==1.4.3
178c179
< google-cloud-storage==1.35.0
---
> google-cloud-storage==1.33.0
184,185c185,186
< google-crc32c==1.1.0
< google-resumable-media==1.2.0
---
> google-crc32c==1.0.0
> google-resumable-media==1.1.0
191c192
< grpcio==1.34.0
---
> grpcio==1.33.2
196,197c197,198
< humanize==3.2.0
< hvac==0.10.6
---
> humanize==3.1.0
> hvac==0.10.5
212c213
< isort==5.6.4
---
> isort==4.3.21
219,220c220,221
< jsonpatch==1.28
< jsonpickle==1.4.2
---
> jsonpatch==1.27
> jsonpickle==1.4.1
232c233
< libcst==0.3.16
---
> libcst==0.3.14
237c238
< marshmallow==3.10.0
---
> marshmallow==3.9.1
239c240
< mock==4.0.3
---
> mock==4.0.2
245,246c246,247
< msal==1.8.0
< msgpack==1.0.2
---
> msal==1.6.0
> msgpack==1.0.0
250c251
< multidict==5.1.0
---
> multidict==5.0.2
267c268
< packaging==20.8
---
> packaging==20.4
269c270
< pandas==1.1.5
---
> pandas==1.1.4
281c282
< pinotdb==0.3.3
---
> pinotdb==0.1.1
285c286
< pre-commit==2.9.3
---
> pre-commit==2.9.2
290c291
< proto-plus==1.13.0
---
> proto-plus==1.11.0
292c293
< psutil==5.8.0
---
> psutil==5.7.3
296,297c297,298
< py==1.10.0
< pyOpenSSL==20.0.1
---
> py==1.9.0
> pyOpenSSL==19.1.0
306,307c307,308
< pydruid==0.6.2
< pyenchant==3.2.0
---
> pydruid==0.6.1
> pyenchant==3.1.1
311,312c312,313
< pylint==2.6.0
< pymongo==3.11.2
---
> pylint==2.5.3
> pymongo==3.11.1
324,325c325,326
< pytest-xdist==2.2.0
< pytest==6.2.1
---
> pytest-xdist==2.1.0
> pytest==6.1.2
355c356
< sendgrid==6.4.8
---
> sendgrid==6.4.7
357,358c358,359
< sentry-sdk==0.19.5
< setproctitle==1.2.1
---
> sentry-sdk==0.19.4
> setproctitle==1.2
364c365
< snowflake-connector-python==2.3.7
---
> snowflake-connector-python==2.3.6
367c368
< soupsieve==2.1
---
> soupsieve==2.0.1
391c392
< tableauserverclient==0.14.1
---
> tableauserverclient==0.14.0
396c397
< testfixtures==6.17.0
---
> testfixtures==6.15.0
404c405
< tqdm==4.54.1
---
> tqdm==4.54.0
414c415
< vertica-python==1.0.1
---
> vertica-python==1.0.0
416c417
< virtualenv==20.2.2
---
> virtualenv==20.2.1
424c425
< yandexcloud==0.64.0
---
> yandexcloud==0.61.0
```
We probably wouldn't prioritize this, but we want updates for snowflake-connector-python as we have ordered a new update to this library that will include other critical changes that prevent the use of snowflake in Airflow 2.0.
See: https://github.com/apache/airflow/issues/12881
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735277308
BTW. As the old saying is "There are only two hard things in computer science, naming conventions, cache invalidation, and off-by-one mistakes". Cleaarly we have cache-invalidation problem here.
Ah yeah. "And DNS, it's always DNS" says @ashb
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735208078
> HI @potiuk , I'm not sure why the azure-blob-storage is not getting installed, despite that I added it to setup.py
In this case you need "upgrade to latest dependencies" label and rebase it . I just added the label. please rebase to the the latest master :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#discussion_r528199888
##########
File path: airflow/providers/microsoft/azure/hooks/wasb.py
##########
@@ -16,94 +16,145 @@
# specific language governing permissions and limitations
# under the License.
#
-"""
-This module contains integration with Azure Blob Storage.
+"""This module contains integration with Azure Blob Storage."""
-It communicate via the Window Azure Storage Blob protocol. Make sure that a
-Airflow connection of type `wasb` exists. Authorization can be done by supplying a
-login (=Storage account name) and password (=KEY), or login and SAS token in the extra
-field (see connection `wasb_default` for an example).
-"""
-from azure.storage.blob import BlockBlobService
+from typing import Any, Dict, List, Optional
+
+from azure.core.exceptions import ResourceExistsError, ResourceNotFoundError
+from azure.identity import ClientSecretCredential
+from azure.storage.blob import BlobClient, BlobServiceClient, ContainerClient, StorageStreamDownloader
from airflow.exceptions import AirflowException
from airflow.hooks.base_hook import BaseHook
class WasbHook(BaseHook):
"""
- Interacts with Azure Blob Storage through the ``wasb://`` protocol.
-
- These parameters have to be passed in Airflow Data Base: account_name and account_key.
-
- Additional options passed in the 'extra' field of the connection will be
- passed to the `BlockBlockService()` constructor. For example, authenticate
- using a SAS token by adding {"sas_token": "YOUR_TOKEN"}.
+ Interacts with Azure Blob Storage.
:param wasb_conn_id: Reference to the wasb connection.
:type wasb_conn_id: str
+ :param public_read: Whether an anonymous public read access should be used. default is False
+ :type public_read: bool
"""
- def __init__(self, wasb_conn_id: str = 'wasb_default') -> None:
+ def __init__(self, wasb_conn_id: str = 'wasb_default', public_read: bool = False) -> None:
super().__init__()
self.conn_id = wasb_conn_id
+ self.public_read = public_read
self.connection = self.get_conn()
- def get_conn(self) -> BlockBlobService:
- """Return the BlockBlobService object."""
+ def get_conn(self) -> BlobServiceClient:
+ """Return the BlobServiceClient object."""
conn = self.get_connection(self.conn_id)
- service_options = conn.extra_dejson
- return BlockBlobService(account_name=conn.login, account_key=conn.password, **service_options)
+ extra = conn.extra_dejson
- def check_for_blob(self, container_name, blob_name, **kwargs):
+ if self.public_read:
+ # Here we use anonymous public read
+ # more info
+ # https://docs.microsoft.com/en-us/azure/storage/blobs/storage-manage-access-to-resources
+ return BlobServiceClient(account_url=conn.host)
+ if extra.get('connection_string'):
+ # connection_string auth takes priority
+ return BlobServiceClient.from_connection_string(extra.get('connection_string'))
+ if extra.get('shared_access_key'):
+ # using shared access key
+ return BlobServiceClient(account_url=conn.host, credential=extra.get('shared_access_key'))
+ if extra.get('tenant_id'):
+ # use Active Directory auth
+ app_id = conn.login
+ app_secret = conn.password
+ token_credential = ClientSecretCredential(extra.get('tenant_id'), app_id, app_secret)
+ return BlobServiceClient(account_url=conn.host, credential=token_credential)
+ if extra.get('sas_token'):
+ return BlobServiceClient(account_url=extra.get('sas_token'))
+ else:
+ raise AirflowException('Unknown connection type')
+
+ def _get_container_client(self, container_name: str) -> ContainerClient:
"""
- Check if a blob exists on Azure Blob Storage.
+ Instantiates a container client
+
+ :param container_name: The name of the container
+ :type container_name: str
+ :return: ContainerClient
+ """
+ return self.connection.get_container_client(container_name)
+
+ def _get_blob_client(self, container_name: str, blob_name: str) -> BlobClient:
+ """
+ Instantiates a blob client
+ :param container_name: The name of the blob container
+ :type container_name: str
+ :param blob_name: The name of the blob. This needs not be existing
+ :type blob_name: str
+ """
+ container_client = self.create_container(container_name)
+ return container_client.get_blob_client(blob_name)
+ def check_for_blob(self, container_name: str, blob_name: str, **kwargs) -> bool:
+ """
+ Check if a blob exists on Azure Blob Storage.
:param container_name: Name of the container.
:type container_name: str
:param blob_name: Name of the blob.
:type blob_name: str
- :param kwargs: Optional keyword arguments that
- `BlockBlobService.exists()` takes.
+ :param kwargs: Optional keyword arguments for `BlobClient.get_blob_properties` takes.
:type kwargs: object
:return: True if the blob exists, False otherwise.
:rtype: bool
"""
- return self.connection.exists(container_name, blob_name, **kwargs)
+ try:
+ self._get_blob_client(container_name, blob_name).get_blob_properties(**kwargs)
+ except ResourceNotFoundError:
+ return False
+ return True
- def check_for_prefix(self, container_name: str, prefix: str, **kwargs) -> bool:
+ def check_for_prefix(self, container_name: str, prefix: str, **kwargs):
"""
Check if a prefix exists on Azure Blob storage.
:param container_name: Name of the container.
:type container_name: str
:param prefix: Prefix of the blob.
:type prefix: str
- :param kwargs: Optional keyword arguments that
- `BlockBlobService.list_blobs()` takes.
+ :param kwargs: Optional keyword arguments that ContainerClient.walk_blobs takes
:type kwargs: object
:return: True if blobs matching the prefix exist, False otherwise.
:rtype: bool
"""
- matches = self.connection.list_blobs(container_name, prefix, num_results=1, **kwargs)
- return len(list(matches)) > 0
+ blobs = self.get_blobs_list(container_name=container_name, prefix=prefix, **kwargs)
+ return len(blobs) > 0
- def get_blobs_list(self, container_name: str, prefix: str, **kwargs) -> list:
+ def get_blobs_list(
+ self,
+ container_name: str,
+ prefix: Optional[str] = None,
+ include: Optional[List[str]] = None,
+ delimiter: Optional[str] = '/',
+ **kwargs,
+ ) -> List:
"""
- Return a list of blobs from path defined in prefix param
+ List blobs in a given container
- :param container_name: Name of the container.
+ :param container_name: The name of the container
:type container_name: str
- :param prefix: Prefix of the blob.
+ :param prefix: Filters the results to return only blobs whose names
+ begin with the specified prefix.
:type prefix: str
- :param kwargs: Optional keyword arguments that
- `BlockBlobService.list_blobs()` takes (num_results, include,
- delimiter, marker, timeout)
- :type kwargs: object
- :return: List of blobs.
- :rtype: list(azure.storage.common.models.ListGenerator)
+ :param include: Specifies one or more additional datasets to include in the
+ response. Options include: 'snapshots', 'metadata', 'uncommittedblobs',
Review comment:
```suggestion
response. Options include: ``snapshots``, ``metadata``, ``uncommittedblobs``,
```
This means that these words need not be in the dictionary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-765983479
Merging it !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735377809
>Can you please link to the build? I'd love to take a look.
I can't find the build. I think the recent pushes I made removed it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735209110
> This is a new feature added last week.
Oh, cool!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-749821255
[The Workflow run](https://github.com/apache/airflow/actions/runs/439098766) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy edited a comment on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ephraimbuddy edited a comment on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735428805
> But I believe that one worked :)
Yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-752951754
Nice. Will rebase. Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#discussion_r528199809
##########
File path: airflow/providers/microsoft/azure/hooks/wasb.py
##########
@@ -16,94 +16,145 @@
# specific language governing permissions and limitations
# under the License.
#
-"""
-This module contains integration with Azure Blob Storage.
+"""This module contains integration with Azure Blob Storage."""
-It communicate via the Window Azure Storage Blob protocol. Make sure that a
-Airflow connection of type `wasb` exists. Authorization can be done by supplying a
-login (=Storage account name) and password (=KEY), or login and SAS token in the extra
-field (see connection `wasb_default` for an example).
-"""
-from azure.storage.blob import BlockBlobService
+from typing import Any, Dict, List, Optional
+
+from azure.core.exceptions import ResourceExistsError, ResourceNotFoundError
+from azure.identity import ClientSecretCredential
+from azure.storage.blob import BlobClient, BlobServiceClient, ContainerClient, StorageStreamDownloader
from airflow.exceptions import AirflowException
from airflow.hooks.base_hook import BaseHook
class WasbHook(BaseHook):
"""
- Interacts with Azure Blob Storage through the ``wasb://`` protocol.
-
- These parameters have to be passed in Airflow Data Base: account_name and account_key.
-
- Additional options passed in the 'extra' field of the connection will be
- passed to the `BlockBlockService()` constructor. For example, authenticate
- using a SAS token by adding {"sas_token": "YOUR_TOKEN"}.
+ Interacts with Azure Blob Storage.
:param wasb_conn_id: Reference to the wasb connection.
:type wasb_conn_id: str
+ :param public_read: Whether an anonymous public read access should be used. default is False
+ :type public_read: bool
"""
- def __init__(self, wasb_conn_id: str = 'wasb_default') -> None:
+ def __init__(self, wasb_conn_id: str = 'wasb_default', public_read: bool = False) -> None:
super().__init__()
self.conn_id = wasb_conn_id
+ self.public_read = public_read
self.connection = self.get_conn()
- def get_conn(self) -> BlockBlobService:
- """Return the BlockBlobService object."""
+ def get_conn(self) -> BlobServiceClient:
+ """Return the BlobServiceClient object."""
conn = self.get_connection(self.conn_id)
- service_options = conn.extra_dejson
- return BlockBlobService(account_name=conn.login, account_key=conn.password, **service_options)
+ extra = conn.extra_dejson
- def check_for_blob(self, container_name, blob_name, **kwargs):
+ if self.public_read:
+ # Here we use anonymous public read
+ # more info
+ # https://docs.microsoft.com/en-us/azure/storage/blobs/storage-manage-access-to-resources
+ return BlobServiceClient(account_url=conn.host)
+ if extra.get('connection_string'):
+ # connection_string auth takes priority
+ return BlobServiceClient.from_connection_string(extra.get('connection_string'))
+ if extra.get('shared_access_key'):
+ # using shared access key
+ return BlobServiceClient(account_url=conn.host, credential=extra.get('shared_access_key'))
+ if extra.get('tenant_id'):
+ # use Active Directory auth
+ app_id = conn.login
+ app_secret = conn.password
+ token_credential = ClientSecretCredential(extra.get('tenant_id'), app_id, app_secret)
+ return BlobServiceClient(account_url=conn.host, credential=token_credential)
+ if extra.get('sas_token'):
+ return BlobServiceClient(account_url=extra.get('sas_token'))
+ else:
+ raise AirflowException('Unknown connection type')
+
+ def _get_container_client(self, container_name: str) -> ContainerClient:
"""
- Check if a blob exists on Azure Blob Storage.
+ Instantiates a container client
+
+ :param container_name: The name of the container
+ :type container_name: str
+ :return: ContainerClient
+ """
+ return self.connection.get_container_client(container_name)
+
+ def _get_blob_client(self, container_name: str, blob_name: str) -> BlobClient:
+ """
+ Instantiates a blob client
+ :param container_name: The name of the blob container
+ :type container_name: str
+ :param blob_name: The name of the blob. This needs not be existing
+ :type blob_name: str
+ """
+ container_client = self.create_container(container_name)
+ return container_client.get_blob_client(blob_name)
+ def check_for_blob(self, container_name: str, blob_name: str, **kwargs) -> bool:
+ """
+ Check if a blob exists on Azure Blob Storage.
:param container_name: Name of the container.
Review comment:
```suggestion
:param container_name: Name of the container.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-765791111
[The Workflow run](https://github.com/apache/airflow/actions/runs/504912767) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-765624818
Some static checks are failing on this one @ephraimbuddy
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#discussion_r528200052
##########
File path: airflow/providers/microsoft/azure/hooks/wasb.py
##########
@@ -189,22 +328,16 @@ def delete_file(
:param ignore_if_missing: if True, then return success even if the
blob does not exist.
:type ignore_if_missing: bool
- :param kwargs: Optional keyword arguments that
- `BlockBlobService.create_blob_from_path()` takes.
+ :param kwargs: Optional keyword arguments that `ContainerClient.delete_blobs()` takes.
Review comment:
```suggestion
:param kwargs: Optional keyword arguments that ``ContainerClient.delete_blobs()`` takes.
```
##########
File path: docs/spelling_wordlist.txt
##########
@@ -1297,6 +1297,7 @@ uid
umask
un
unarchived
+uncommittedblobs
Review comment:
```suggestion
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-749921653
> When I ran breeze locally, I found that azure-storage 0.36 is still getting installed
Yes. We have extra protection to prevent image damage. In most cases, the constraints file is used to build the image, which contains a set of tested libraries. But we would like to update this file automatically sometimes, so we need the normal pip install to install the correct dependencies.
More info:
http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/installation.html
https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pinned-constraint-files
Recently, snowflake-connector-python dropped support for old azure library, so now snowflake and azure can't be installed, because pip will have trouble finding dependencies.
See: https://github.com/apache/airflow/pull/13195
In this PR, I am trying to improve compatibility with newer versions of Pinot and the pinot-python library, so I added the label "upgrade to newer dependencies" which means the image will be built without constraint, so the dependency installation may fail and it happened now. Several libraries have been updated and unfortunately, these dependencies do not work.
Here I have compiled a list of the differences between installing with and without a constraints file. Most are safe and will be updated soon in constraints.txt as well, but we have a few changes we will add to setup.py/setup.cfg, so users will be limited in using these libraries.
```
diff <(pip freeze | sort) <(curl https://raw.githubusercontent.com/apache/airflow/constraints-master/constraints-3.6.txt | sort)
< -e /opt/airflow
19c18
< JPype1==1.2.0
---
> JPype1==1.1.2
30c29
< Pygments==2.7.3
---
> Pygments==2.7.2
33,35c32,34
< SQLAlchemy==1.3.22
< Sphinx==3.4.0
< Unidecode==1.1.2
---
> SQLAlchemy==1.3.20
> Sphinx==3.3.1
> Unidecode==1.1.1
56c55
< aws-sam-translator==1.33.0
---
> aws-sam-translator==1.31.0
58c57
< azure-batch==10.0.0
---
> azure-batch==9.0.0
65c64
< azure-keyvault-keys==4.3.1
---
> azure-keyvault-keys==4.3.0
90c89
< cachetools==4.2.0
---
> cachetools==4.1.1
94c93
< certifi==2020.12.5
---
> certifi==2020.11.8
97c96
< cfn-lint==0.44.1
---
> cfn-lint==0.42.0
109c108
< coverage==5.3.1
---
> coverage==5.3
111c110
< cryptography==3.3.1
---
> cryptography==3.2.1
113,114c112,113
< cx-Oracle==8.1.0
< dask==2020.12.0
---
> cx-Oracle==8.0.1
> dask==2.30.0
133c132
< eventlet==0.30.0
---
> eventlet==0.29.1
135,136c134,135
< facebook-business==9.0.1
< fastavro==1.2.1
---
> facebook-business==9.0.0
> fastavro==1.2.0
142c141,142
< flower==0.9.7
---
> flask-swagger==0.2.13
> flower==0.9.5
144c144,145
< fsspec==0.8.5
---
> fsspec==0.8.4
> funcsigs==1.0.2
152c153
< google-api-core==1.24.1
---
> google-api-core==1.23.0
156c157
< google-auth==1.24.0
---
> google-auth==1.23.0
160,161c161,162
< google-cloud-bigquery==2.6.1
< google-cloud-bigtable==1.6.1
---
> google-cloud-bigquery==2.4.0
> google-cloud-bigtable==1.6.0
163c164
< google-cloud-core==1.5.0
---
> google-cloud-core==1.4.3
178c179
< google-cloud-storage==1.35.0
---
> google-cloud-storage==1.33.0
184,185c185,186
< google-crc32c==1.1.0
< google-resumable-media==1.2.0
---
> google-crc32c==1.0.0
> google-resumable-media==1.1.0
191c192
< grpcio==1.34.0
---
> grpcio==1.33.2
196,197c197,198
< humanize==3.2.0
< hvac==0.10.6
---
> humanize==3.1.0
> hvac==0.10.5
212c213
< isort==5.6.4
---
> isort==4.3.21
219,220c220,221
< jsonpatch==1.28
< jsonpickle==1.4.2
---
> jsonpatch==1.27
> jsonpickle==1.4.1
232c233
< libcst==0.3.16
---
> libcst==0.3.14
237c238
< marshmallow==3.10.0
---
> marshmallow==3.9.1
239c240
< mock==4.0.3
---
> mock==4.0.2
245,246c246,247
< msal==1.8.0
< msgpack==1.0.2
---
> msal==1.6.0
> msgpack==1.0.0
250c251
< multidict==5.1.0
---
> multidict==5.0.2
267c268
< packaging==20.8
---
> packaging==20.4
269c270
< pandas==1.1.5
---
> pandas==1.1.4
281c282
< pinotdb==0.3.3
---
> pinotdb==0.1.1
285c286
< pre-commit==2.9.3
---
> pre-commit==2.9.2
290c291
< proto-plus==1.13.0
---
> proto-plus==1.11.0
292c293
< psutil==5.8.0
---
> psutil==5.7.3
296,297c297,298
< py==1.10.0
< pyOpenSSL==20.0.1
---
> py==1.9.0
> pyOpenSSL==19.1.0
306,307c307,308
< pydruid==0.6.2
< pyenchant==3.2.0
---
> pydruid==0.6.1
> pyenchant==3.1.1
311,312c312,313
< pylint==2.6.0
< pymongo==3.11.2
---
> pylint==2.5.3
> pymongo==3.11.1
324,325c325,326
< pytest-xdist==2.2.0
< pytest==6.2.1
---
> pytest-xdist==2.1.0
> pytest==6.1.2
355c356
< sendgrid==6.4.8
---
> sendgrid==6.4.7
357,358c358,359
< sentry-sdk==0.19.5
< setproctitle==1.2.1
---
> sentry-sdk==0.19.4
> setproctitle==1.2
364c365
< snowflake-connector-python==2.3.7
---
> snowflake-connector-python==2.3.6
367c368
< soupsieve==2.1
---
> soupsieve==2.0.1
391c392
< tableauserverclient==0.14.1
---
> tableauserverclient==0.14.0
396c397
< testfixtures==6.17.0
---
> testfixtures==6.15.0
404c405
< tqdm==4.54.1
---
> tqdm==4.54.0
414c415
< vertica-python==1.0.1
---
> vertica-python==1.0.0
416c417
< virtualenv==20.2.2
---
> virtualenv==20.2.1
424c425
< yandexcloud==0.64.0
---
> yandexcloud==0.61.0
```
We probably wouldn't prioritize this, but we want updates for snowflake-connector-python as we have ordered a new update to this library that will include other critical changes that prevent the use of snowflake in Airflow 2.0.
See: https://github.com/apache/airflow/issues/12881
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#discussion_r528199906
##########
File path: airflow/providers/microsoft/azure/hooks/wasb.py
##########
@@ -115,12 +166,11 @@ def load_file(self, file_path: str, container_name: str, blob_name: str, **kwarg
:type container_name: str
:param blob_name: Name of the blob.
:type blob_name: str
- :param kwargs: Optional keyword arguments that
- `BlockBlobService.create_blob_from_path()` takes.
+ :param kwargs: Optional keyword arguments that `BlobClient.upload_blob()` takes.
Review comment:
```suggestion
:param kwargs: Optional keyword arguments that ``BlobClient.upload_blob()`` takes.
```
##########
File path: airflow/providers/microsoft/azure/hooks/wasb.py
##########
@@ -132,12 +182,11 @@ def load_string(self, string_data: str, container_name: str, blob_name: str, **k
:type container_name: str
:param blob_name: Name of the blob.
:type blob_name: str
- :param kwargs: Optional keyword arguments that
- `BlockBlobService.create_blob_from_text()` takes.
+ :param kwargs: Optional keyword arguments that `BlobClient.upload()` takes.
Review comment:
```suggestion
:param kwargs: Optional keyword arguments that ``BlobClient.upload()`` takes.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-767083620
This should be released in 2.0.1, right? @ephraimbuddy @potiuk
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-760200033
[The Workflow run](https://github.com/apache/airflow/actions/runs/485419067) is cancelling this PR. Building images for the PR has failed. Follow the the workflow link to check the reason.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-765869315
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#discussion_r528257169
##########
File path: docs/spelling_wordlist.txt
##########
@@ -1297,6 +1297,7 @@ uid
umask
un
unarchived
+uncommittedblobs
Review comment:
This will work if you write each occurrence of this word as a code literal:
````
A ``uncommiitedblobs`` is an awesome thing
````
````
A uncommiitedblobs is an awesome thing
````
The first example causes no problems. The second will be raise warning by spellcheck.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735378572
I found this from my email https://github.com/apache/airflow/actions/runs/365256644
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-749921653
> When I ran breeze locally, I found that azure-storage 0.36 is still getting installed
Yes. We have extra protection to prevent image damage. In most cases, the constraints file is used to build the image, which contains a set of tested libraries. But we would like to update this file automatically sometimes, so we need the normal pip install to install the correct dependencies.
More info:
http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/installation.html
https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pinned-constraint-files
Recently, snowflake-connector-python dropped support for old azure library, so now snowflake and azure can't be installed, because pip will have trouble finding dependencies.
See: https://github.com/apache/airflow/pull/13195
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-752968853
[The Workflow run](https://github.com/apache/airflow/actions/runs/454631420) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-765810621
I was able to get this to install fine in my machine and I believe it'll also work on the CI. A little addition to the workaround @potiuk added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735302772
> That should be rather easy I think. In this case we need to make a little more changes because we do not handle well "uninstalling of a previously installed packages from cache" but it can be done rather easily:
>
> In the Dockerfile.ci please change those two lines - > change the AIRFLOW_CI_BUILD_EPOCH to 4, and AIRFLOW_PRE_CACHED_PIP_PACKAGES to "false" :
>
> ```
> # By changing the CI build epoch we can force reinstalling Airflow from the current master
> # It can also be overwritten manually by setting the AIRFLOW_CI_BUILD_EPOCH environment variable.
> ARG AIRFLOW_CI_BUILD_EPOCH="3"
> ENV AIRFLOW_CI_BUILD_EPOCH=${AIRFLOW_CI_BUILD_EPOCH}
>
> ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="true"
> ENV AIRFLOW_PRE_CACHED_PIP_PACKAGES=${AIRFLOW_PRE_CACHED_PIP_PACKAGES}
> ```
>
> What those two changes mean:
>
> * increasing the EPOCH causes invalidation of Docker Cache from this place. This means that for your PR the dockerfile will be rebuilt starting from that line.
> * setting AIRFLOW_PRE_CACHED_PIP_PACKAGES to "false" disables the optimisation where we use the previously build images as cache for new builds - thanks to that we save a lot of time for image builds, but whatever was originally installed in the image remains there (including 'azure-storage' package).
>
> If you change both values, your build will run longer, but it will not have the 'azure-storage' package installed at all (since you removed it from the setup.py).
>
> When we see that all that works, We will just leave the EPOCH change. Once your change will be merge to master, all new builds will use the new master version of setup.py and will not install 'azure-storage' once the rebase to master and get your EPOCH change.
>
> I hope it helps , ping me on slaack when you re-rerun it and still have some problems.
I tried this and it didn't work.
I saw that both `azure-storage-blob` and `azure-storage` are in https://raw.githubusercontent.com/apache/airflow/constraints-master/constraints-3.6.txt
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-759522368
@ephraimbuddy Cancelled because of the long wait in the queue :( . I m afraid at least one more rebase will be needed :( sorry.
On a good front https://github.com/apache/airflow/issues/12881 upgrades snowflake and removes some limitation, so we are getting closer.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#discussion_r528199826
##########
File path: airflow/providers/microsoft/azure/hooks/wasb.py
##########
@@ -16,94 +16,145 @@
# specific language governing permissions and limitations
# under the License.
#
-"""
-This module contains integration with Azure Blob Storage.
+"""This module contains integration with Azure Blob Storage."""
-It communicate via the Window Azure Storage Blob protocol. Make sure that a
-Airflow connection of type `wasb` exists. Authorization can be done by supplying a
-login (=Storage account name) and password (=KEY), or login and SAS token in the extra
-field (see connection `wasb_default` for an example).
-"""
-from azure.storage.blob import BlockBlobService
+from typing import Any, Dict, List, Optional
+
+from azure.core.exceptions import ResourceExistsError, ResourceNotFoundError
+from azure.identity import ClientSecretCredential
+from azure.storage.blob import BlobClient, BlobServiceClient, ContainerClient, StorageStreamDownloader
from airflow.exceptions import AirflowException
from airflow.hooks.base_hook import BaseHook
class WasbHook(BaseHook):
"""
- Interacts with Azure Blob Storage through the ``wasb://`` protocol.
-
- These parameters have to be passed in Airflow Data Base: account_name and account_key.
-
- Additional options passed in the 'extra' field of the connection will be
- passed to the `BlockBlockService()` constructor. For example, authenticate
- using a SAS token by adding {"sas_token": "YOUR_TOKEN"}.
+ Interacts with Azure Blob Storage.
:param wasb_conn_id: Reference to the wasb connection.
:type wasb_conn_id: str
+ :param public_read: Whether an anonymous public read access should be used. default is False
+ :type public_read: bool
"""
- def __init__(self, wasb_conn_id: str = 'wasb_default') -> None:
+ def __init__(self, wasb_conn_id: str = 'wasb_default', public_read: bool = False) -> None:
super().__init__()
self.conn_id = wasb_conn_id
+ self.public_read = public_read
self.connection = self.get_conn()
- def get_conn(self) -> BlockBlobService:
- """Return the BlockBlobService object."""
+ def get_conn(self) -> BlobServiceClient:
+ """Return the BlobServiceClient object."""
conn = self.get_connection(self.conn_id)
- service_options = conn.extra_dejson
- return BlockBlobService(account_name=conn.login, account_key=conn.password, **service_options)
+ extra = conn.extra_dejson
- def check_for_blob(self, container_name, blob_name, **kwargs):
+ if self.public_read:
+ # Here we use anonymous public read
+ # more info
+ # https://docs.microsoft.com/en-us/azure/storage/blobs/storage-manage-access-to-resources
+ return BlobServiceClient(account_url=conn.host)
+ if extra.get('connection_string'):
+ # connection_string auth takes priority
+ return BlobServiceClient.from_connection_string(extra.get('connection_string'))
+ if extra.get('shared_access_key'):
+ # using shared access key
+ return BlobServiceClient(account_url=conn.host, credential=extra.get('shared_access_key'))
+ if extra.get('tenant_id'):
+ # use Active Directory auth
+ app_id = conn.login
+ app_secret = conn.password
+ token_credential = ClientSecretCredential(extra.get('tenant_id'), app_id, app_secret)
+ return BlobServiceClient(account_url=conn.host, credential=token_credential)
+ if extra.get('sas_token'):
+ return BlobServiceClient(account_url=extra.get('sas_token'))
+ else:
+ raise AirflowException('Unknown connection type')
+
+ def _get_container_client(self, container_name: str) -> ContainerClient:
"""
- Check if a blob exists on Azure Blob Storage.
+ Instantiates a container client
+
+ :param container_name: The name of the container
+ :type container_name: str
+ :return: ContainerClient
+ """
+ return self.connection.get_container_client(container_name)
+
+ def _get_blob_client(self, container_name: str, blob_name: str) -> BlobClient:
+ """
+ Instantiates a blob client
+ :param container_name: The name of the blob container
+ :type container_name: str
+ :param blob_name: The name of the blob. This needs not be existing
+ :type blob_name: str
+ """
+ container_client = self.create_container(container_name)
+ return container_client.get_blob_client(blob_name)
+ def check_for_blob(self, container_name: str, blob_name: str, **kwargs) -> bool:
+ """
+ Check if a blob exists on Azure Blob Storage.
:param container_name: Name of the container.
:type container_name: str
:param blob_name: Name of the blob.
:type blob_name: str
- :param kwargs: Optional keyword arguments that
- `BlockBlobService.exists()` takes.
+ :param kwargs: Optional keyword arguments for `BlobClient.get_blob_properties` takes.
Review comment:
```suggestion
:param kwargs: Optional keyword arguments for ``BlobClient.get_blob_properties`` takes.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735276782
That should be rather easy I think. In this case we need to make a little more changes because we do not handle well "uninstalling of a previously installed package but it can be done rather easily:
In the Dockerfile.ci please change those two lines - > change the AIRFLOW_CI_BUILD_EPOCH to 4, and AIRFLOW_PRE_CACHED_PIP_PACKAGES to "false" :
```
# By changing the CI build epoch we can force reinstalling Airflow from the current master
# It can also be overwritten manually by setting the AIRFLOW_CI_BUILD_EPOCH environment variable.
ARG AIRFLOW_CI_BUILD_EPOCH="3"
ENV AIRFLOW_CI_BUILD_EPOCH=${AIRFLOW_CI_BUILD_EPOCH}
ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="true"
ENV AIRFLOW_PRE_CACHED_PIP_PACKAGES=${AIRFLOW_PRE_CACHED_PIP_PACKAGES}
```
What those two changes mean:
* increasing the EPOCH causes invalidation of Docker Cache from this place. This means that for your PR the dockerfile will be rebuilt starting from that line.
* setting AIRFLOW_PRE_CACHED_PIP_PACKAGES to "false" disables the optimisation where we use the previously build images as cache for new builds - thanks to that we save a lot of time for image builds, but whatever was originally installed in the image remains there (including 'azure-storage' package).
If you change both values, your build will run longer, but it will not have the 'azure-storage' package installed at all (since you removed it from the setup.py).
When we see that all that works, We will just leave the EPOCH change. Once your change will be merge to master, all new builds will use the new master version of setup.py and will not install 'azure-storage' once the rebase to master and get your EPOCH change.
I hope it helps , ping me on slaack when you re-rerun it and still have some problems.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #12188: Upgrade azure blob to v12
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-735119637
HI @potiuk , I'm not sure why the azure-blob-storage is not getting installed, despite that I added it to setup.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org