You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/02/17 04:10:35 UTC

[GitHub] [airflow] Swalloow opened a new issue #14266: Azure Provider related WARNING message in webserver (version 2.0.1)

Swalloow opened a new issue #14266:
URL: https://github.com/apache/airflow/issues/14266


   <!--
   
   Welcome to Apache Airflow!  For a smooth issue process, try to answer the following questions.
   Don't worry if they're not all applicable; just try to include what you can :-)
   
   If you need to include code snippets or logs, please put them in fenced code
   blocks.  If they're super-long, please use the details tag like
   <details><summary>super-long log</summary> lots of stuff </details>
   
   Please delete these comment blocks before submitting the issue.
   
   -->
   
   <!--
   
   IMPORTANT!!!
   
   PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
   NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
   
   PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
   
   Please complete the next sections or the issue will be closed.
   These questions are the first thing we need to know to understand the context.
   
   -->
   
   **Apache Airflow version**: 2.0.1
   
   
   **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): 1.18
   
   **Environment**: Official Helm Chart with CeleryExecutor
   
   - **Cloud provider or hardware configuration**: AWS EKS
   - **OS** (e.g. from /etc/os-release):
   - **Kernel** (e.g. `uname -a`):
   - **Install tools**:
   - **Others**:
   
   **What happened**:
   
   <!-- (please include exact error messages if you can) -->
   
   When updated to `apache/airflow:2.0.1-python3.7` image, a WARNING message occurs in the webserver.
   
   ```
   [2021-02-17 04:06:47,339] {providers_manager.py:299} 
   WARNING - Exception when importing 'airflow.providers.microsoft.azure.hooks.wasb.WasbHook' from 
   'apache-airflow-providers-microsoft-azure' package: No module named 'azure.storage.blob'
   ```
   
   **What you expected to happen**:
   
   <!-- What do you think went wrong? -->
   
   **How to reproduce it**:
   <!---
   
   As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags.
   
   If you are using kubernetes, please attempt to recreate the issue using minikube or kind.
   
   ## Install minikube/kind
   
   - Minikube https://minikube.sigs.k8s.io/docs/start/
   - Kind https://kind.sigs.k8s.io/docs/user/quick-start/
   
   If this is a UI bug, please provide a screenshot of the bug or a link to a youtube video of the bug in action
   
   You can include images using the .md style of
   ![alt text](http://url/to/img.png)
   
   To record a screencast, mac users can use QuickTime and then create an unlisted youtube video with the resulting .mov file.
   
   --->
   
   
   **Anything else we need to know**:
   
   <!--
   
   How often does this problem occur? Once? Every time etc?
   
   Any relevant logs to include? Put them here in side a detail tag:
   <details><summary>x.log</summary> lots of stuff </details>
   
   -->
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] pl77 edited a comment on issue #14266: Azure Provider related WARNING in webserver (version 2.0.1)

Posted by GitBox <gi...@apache.org>.
pl77 edited a comment on issue #14266:
URL: https://github.com/apache/airflow/issues/14266#issuecomment-796923376


   For those who might be looking for a simple way to implement these suggestions, I tried solutions from both @omarish and @potiuk and they both worked fine.  Easiest way to try them was create a blank text file named, as an example, `Dockerfile-potiuk` and inside place the commands:
   
   ```
   FROM apache/airflow:2.0.1
   RUN pip install apache-airflow-providers-microsoft-azure==1.2.0rc1
   ```
   
   Then, in the same directory you saved that file, run:
   
   ```bash
   docker build -f ./Dockerfile-potiuk -t apache/airflow:2.0.1 .
   ```
   
   If you save with the same tag as the existing file you don't have to change your `docker-compose.yml` files and when 2.0.2 is released it'll supersede this monkey patch when you do your regular `docker-compose pull` when upgrading (of course, that's if your image is specified as `:latest` in your compose file). 
   
   I'm sure everyone using airflow is advanced enough to know this already but figured I'd post it just in case someone was struggling.
   
   @potiuk :  the rc1 candidate seemed to install just fine but I didn't have an azure instance to test it against, unfortunately. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] JasonPierce edited a comment on issue #14266: Azure Provider related WARNING in webserver (version 2.0.1)

Posted by GitBox <gi...@apache.org>.
JasonPierce edited a comment on issue #14266:
URL: https://github.com/apache/airflow/issues/14266#issuecomment-782286146


   Happening in `apache/airflow:2.0.1-python3.8`, as well.
   
   Workaround:
   
   1. Start Airflow: `docker-compose up`
   2. Open bash in webserver container: `docker exec -it airflow_airflow-webserver_1 /bin/bash`
       - use `docker ps` to find your webserver container name if it differs from `airflow_airflow-webserver_1`
   3. Install azure-storage-blob (thanks @kaxil) in webserver container: `pip uninstall --yes azure-storage && pip install -U azure-storage-blob apache-airflow-providers-microsoft-azure==1.1.0`
   
   Didn't have to restart webserver, it just started working after running steps 1-3. YMMV.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] pl77 commented on issue #14266: Azure Provider related WARNING in webserver (version 2.0.1)

Posted by GitBox <gi...@apache.org>.
pl77 commented on issue #14266:
URL: https://github.com/apache/airflow/issues/14266#issuecomment-796923376


   For those who might be looking for a simple way to implement these suggestions, I tried solutions from both @omarish and @karakanb and they both worked fine.  Easiest way to try them was create a blank text file named, as an example, `Dockerfile-karaknb` and inside place the commands:
   
   ```
   FROM apache/airflow:2.0.1
   RUN pip install apache-airflow-providers-microsoft-azure==1.2.0rc1
   ```
   
   Then, in the same directory you saved that file, run:
   
   ```bash
   docker build -f ./Dockerfile-karakanb -t apache/airflow:2.0.1 .
   ```
   
   If you save with the same tag as the existing file you don't have to change your `docker-compose.yml` files and when 2.0.2 is released it'll supersede this monkey patch when you do your regular `docker-compose pull` when upgrading (of course, that's if your image is specified as `:latest` in your compose file). 
   
   I'm sure everyone using airflow is advanced enough to know this already but figured I'd post it just in case someone was struggling.
   
   And the rc1 candidate seemed to install just fine but I didn't have an azure instance to test it against, unfortunately. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] don4of4 commented on issue #14266: Azure Provider related WARNING in webserver (version 2.0.1)

Posted by GitBox <gi...@apache.org>.
don4of4 commented on issue #14266:
URL: https://github.com/apache/airflow/issues/14266#issuecomment-782996053


   Experiencing this as well -- obviously running commands in the container isn't great.  Is this fixed in a more recent version?
   
   Also seeing: ` Exception when importing 'airflow.providers.google.cloud.hooks.compute_ssh.ComputeEngineSSHHook' from 'apache-airflow-providers-google' package: cannot import name 'ImportSshPublicKeyResponse' from 'google.cloud.oslogin_v1'`


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #14266: Azure Provider related WARNING in webserver (version 2.0.1)

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #14266:
URL: https://github.com/apache/airflow/issues/14266#issuecomment-787129055


   @karakanb -> correct - that was a problem with dependencies in the image. But it will be fixed in the upcoming bugfix releases of the provider - first, and of the airflow version and image (as second step)
   
   Just to clarify the original issue:
   
   The 2.0.1 image was prepared with the provider packages released at that time (`apache-airflow-providers-microsoft-azure` https://pypi.org/project/apache-airflow-providers-microsoft-azure/1.1.0/ - what you can see there is that `azure-storage-blob` was not a dependency of the `microsoft-azure` provider.
   
   This is fixed in the upcoming 1.2.0 provider: https://pypi.org/project/apache-airflow-providers-microsoft-azure/1.2.0rc1/ (released for voting today). You will see there that `azure-storage-blob` is a dependency there.
   
   The way to mitigate the issue:
   
   * install azure-storage-blob manually in your "extended image" -> @omarish's solution works.
   * wait until 1.2.0 provider package for `microsoft-azure` gets released (early next week) and install it (similar way as above). You can also install it by specifying the rc version if you would be so kind to test it:
   
   ```
   FROM apache/airflow:2.0.1
   RUN pip install apache-airflow-providers-microsoft-azure==1.2.0rc1
   ```
   *  when we release 2.0.2 version of Airflow (week-two from now I guess), the released packages will be automatically used (including the 1.2.0 of microsoft-azure) so 2.0.2 should not have this problem any more.
   
   Sorry for the confusion. The azure dependencies were quite a bit of a problem (due to old/new libraries used for storage by different providers) - but as of 2.0.2, those problems should be gone forever :crossed_fingers: 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] JasonPierce edited a comment on issue #14266: Azure Provider related WARNING in webserver (version 2.0.1)

Posted by GitBox <gi...@apache.org>.
JasonPierce edited a comment on issue #14266:
URL: https://github.com/apache/airflow/issues/14266#issuecomment-782286146


   Happening in `apache/airflow:2.0.1-python3.8`, as well.
   
   Workaround:
   
   1. Start Airflow: `docker-compose up`
   2. Open bash in webserver container: `docker exec -it airflow_airflow-webserver_1 /bin/bash` (use `docker ps` to find your webserver container name)
   3. Install azure-storage-blob (thanks @kaxil): `pip uninstall --yes azure-storage && pip install -U azure-storage-blob apache-airflow-providers-microsoft-azure==1.1.0`
   
   Didn't have to restart webserver, it just started working after running steps 1-3. YMMV.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] JasonPierce edited a comment on issue #14266: Azure Provider related WARNING in webserver (version 2.0.1)

Posted by GitBox <gi...@apache.org>.
JasonPierce edited a comment on issue #14266:
URL: https://github.com/apache/airflow/issues/14266#issuecomment-782286146


   Happening in `apache/airflow:2.0.1-python3.8`, as well.
   
   Workaround:
   
   1. Start Airflow: `docker-compose up`
   2. Open bash in webserver container: `docker exec -it airflow_airflow-webserver_1 /bin/bash` (use `docker ps` to find your webserver container name)
   3. Install azure-storage-blob (thanks @kaxil) in webserver container: `pip uninstall --yes azure-storage && pip install -U azure-storage-blob apache-airflow-providers-microsoft-azure==1.1.0`
   
   Didn't have to restart webserver, it just started working after running steps 1-3. YMMV.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #14266: Azure Provider related WARNING in webserver (version 2.0.1)

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #14266:
URL: https://github.com/apache/airflow/issues/14266


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #14266: Azure Provider related WARNING in webserver (version 2.0.1)

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #14266:
URL: https://github.com/apache/airflow/issues/14266#issuecomment-783474808


   This is totally different problem. You have - most likely - wrong dependency installed you likely have an odll. Providers 2.0.0 have different set of dependencies. See  http://airflow.apache.org/docs/apache-airflow-providers-google/stable/index.html#breaking-changes
   
   Running 'pip check' should tell you if you have wrong set of dependencies.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] karakanb commented on issue #14266: Azure Provider related WARNING in webserver (version 2.0.1)

Posted by GitBox <gi...@apache.org>.
karakanb commented on issue #14266:
URL: https://github.com/apache/airflow/issues/14266#issuecomment-787119536


   @potiuk The problem seems to be with the Docker image itself, meaning that there is no dependency installed from the user's perspective. What you are saying would be correct if the user was trying to run them with their local dependencies, but in the case of the provided Docker example there is simply nothing installed from the host device, therefore this needs to be fixed in the image level as far as I can see.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] JasonPierce edited a comment on issue #14266: Azure Provider related WARNING in webserver (version 2.0.1)

Posted by GitBox <gi...@apache.org>.
JasonPierce edited a comment on issue #14266:
URL: https://github.com/apache/airflow/issues/14266#issuecomment-782286146


   Happening in `apache/airflow:2.0.1-python3.8`, as well.
   
   Workaround:
   
   1. Start Airflow: `docker-compose up`
   2. Open bash in webserver container: `docker exec -it airflow_airflow-webserver_1 /bin/bash` (use `docker ps` to find your webserver container name if it differs from `airflow_airflow-webserver_1`)
   3. Install azure-storage-blob (thanks @kaxil) in webserver container: `pip uninstall --yes azure-storage && pip install -U azure-storage-blob apache-airflow-providers-microsoft-azure==1.1.0`
   
   Didn't have to restart webserver, it just started working after running steps 1-3. YMMV.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] pl77 edited a comment on issue #14266: Azure Provider related WARNING in webserver (version 2.0.1)

Posted by GitBox <gi...@apache.org>.
pl77 edited a comment on issue #14266:
URL: https://github.com/apache/airflow/issues/14266#issuecomment-796923376


   For those who might be looking for a simple way to implement these suggestions, I tried solutions from both @omarish and @potiuk and they both worked fine.  Easiest way to try them was create a blank text file named, as an example, `Dockerfile-karaknb` and inside place the commands:
   
   ```
   FROM apache/airflow:2.0.1
   RUN pip install apache-airflow-providers-microsoft-azure==1.2.0rc1
   ```
   
   Then, in the same directory you saved that file, run:
   
   ```bash
   docker build -f ./Dockerfile-potiuk -t apache/airflow:2.0.1 .
   ```
   
   If you save with the same tag as the existing file you don't have to change your `docker-compose.yml` files and when 2.0.2 is released it'll supersede this monkey patch when you do your regular `docker-compose pull` when upgrading (of course, that's if your image is specified as `:latest` in your compose file). 
   
   I'm sure everyone using airflow is advanced enough to know this already but figured I'd post it just in case someone was struggling.
   
   @potiuk :  the rc1 candidate seemed to install just fine but I didn't have an azure instance to test it against, unfortunately. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] JasonPierce commented on issue #14266: Azure Provider related WARNING in webserver (version 2.0.1)

Posted by GitBox <gi...@apache.org>.
JasonPierce commented on issue #14266:
URL: https://github.com/apache/airflow/issues/14266#issuecomment-782286146


   Happening in `apache/airflow:2.0.1-python3.8`, as well.
   
   Workaround (thanks @kaxil):
   
   1. Start Airflow: `docker-compose up`
   2. Open bash in webserver container: `docker exec -it airflow_airflow-webserver_1 /bin/bash` (use `docker ps` to find your webserver container name)
   3. Install azure-storage-blob: `pip uninstall --yes azure-storage && pip install -U azure-storage-blob apache-airflow-providers-microsoft-azure==1.1.0`
   
   Didn't have to restart webserver, it just started working after running steps 1-3. YMMV.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil commented on issue #14266: Azure Provider related WARNING in webserver (version 2.0.1)

Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #14266:
URL: https://github.com/apache/airflow/issues/14266#issuecomment-780761542


   Try the following command inside the container:
   
   ```
   pip uninstall  --yes azure-storage && pip install -U azure-storage-blob apache-airflow-providers-microsoft-azure==1.1.0
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] omarish commented on issue #14266: Azure Provider related WARNING in webserver (version 2.0.1)

Posted by GitBox <gi...@apache.org>.
omarish commented on issue #14266:
URL: https://github.com/apache/airflow/issues/14266#issuecomment-786298240


   I worked around this by adding another Dockerfile with the pip install:
   
   ```dockerfile
   FROM apache/airflow:2.0.1
   RUN pip uninstall  --yes azure-storage && pip install -U azure-storage-blob apache-airflow-providers-microsoft-azure==1.1.0
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #14266: Azure Provider related WARNING in webserver (version 2.0.1)

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #14266:
URL: https://github.com/apache/airflow/issues/14266#issuecomment-783475272


   Seems that this issue can be closed. I do not think we can do much more here :(


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org