You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/13 15:21:57 UTC

[GitHub] [airflow] ShawnMcGough opened a new issue #14758: Example DAG for New Azure Data Factory Hooks

ShawnMcGough opened a new issue #14758:
URL: https://github.com/apache/airflow/issues/14758


   **Description**
   Add an example DAG to the [example_dags ](https://github.com/apache/airflow/tree/master/airflow/providers/microsoft/azure/example_dags) directory of the azure providers.
   
   **Use case / motivation**
   The [Data Factory hooks](https://github.com/apache/airflow/blob/99c74968180ab7bc6d7152ec4233440b62a07969/airflow/providers/microsoft/azure/hooks/azure_data_factory.py) are new with [apache-airflow-providers-microsoft-azure 1.2.0](https://pypi.org/project/apache-airflow-providers-microsoft-azure/1.2.0/), but there are no examples or documentation on how to use them.
   
   As a new user, it is helpful to have a known working example, especially for troubleshooting config/setup (missing providers, etc).
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ShawnMcGough commented on issue #14758: Example DAG for New Azure Data Factory Hooks

Posted by GitBox <gi...@apache.org>.
ShawnMcGough commented on issue #14758:
URL: https://github.com/apache/airflow/issues/14758#issuecomment-807319061


   @gmcrocetti I did see that error, but if I recall correctly, that error is resolved after installing the recently released version 1.2 of  [apache-airflow-providers-microsoft-azure](https://pypi.org/project/apache-airflow-providers-microsoft-azure/1.2.0).
   
   I do get the following errors on `airflow-web`. I've restarted the pod numerous times. As mentioned above, using the `Azure` type does work.
   
   ```
   [2021-03-25 18:49:36,501] {providers_manager.py:249} WARNING - The '<class 'airflow.providers.microsoft.azure.hooks.azure_data_factory.AzureDataFactoryHook'>' is missing conn_type attribute and cannot be registered
   [2021-03-25 18:49:36,501] {providers_manager.py:249} WARNING - The '<class 'airflow.providers.microsoft.azure.hooks.azure_data_factory.AzureDataFactoryHook'>' is missing conn_name_attr attribute and cannot be registered
   [2021-03-25 18:49:36,501] {providers_manager.py:249} WARNING - The '<class 'airflow.providers.microsoft.azure.hooks.azure_data_factory.AzureDataFactoryHook'>' is missing hook_name attribute and cannot be registered
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on issue #14758: Example DAG for New Azure Data Factory Hooks

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #14758:
URL: https://github.com/apache/airflow/issues/14758#issuecomment-798685497


   @flvndh, I have assigned you to this ticket. 👍


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] gmcrocetti commented on issue #14758: Example DAG for New Azure Data Factory Hooks

Posted by GitBox <gi...@apache.org>.
gmcrocetti commented on issue #14758:
URL: https://github.com/apache/airflow/issues/14758#issuecomment-805248640


   Hey @ShawnMcGough . I'm worried about the lack of `Azure Data Factory` in this drop down menu. Do you see any exception being raised when you click in this menu ? The freshly inserted [hook_name](https://github.com/apache/airflow/blob/a7e144bec855f6ccf0fa5ae8447894195ffe170f/airflow/providers/microsoft/azure/hooks/azure_data_factory.py#L83) should've been there. @ephraimbuddy  @turbaszek, is there something I'm missing ? ProvidersManager seems straightforward with this return.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal closed issue #14758: Example DAG for New Azure Data Factory Hooks

Posted by GitBox <gi...@apache.org>.
eladkal closed issue #14758:
URL: https://github.com/apache/airflow/issues/14758


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] tsaaii commented on issue #14758: Example DAG for New Azure Data Factory Hooks

Posted by GitBox <gi...@apache.org>.
tsaaii commented on issue #14758:
URL: https://github.com/apache/airflow/issues/14758#issuecomment-1033240492


   I tested this with Azure data factory drop down and it works.
   
   I have a question here.
   
   I have my target pipeline in a folder and that is in a feature branch on my ADF.
   
   How do I run this?
   
   Where should I specify here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] gmcrocetti edited a comment on issue #14758: Example DAG for New Azure Data Factory Hooks

Posted by GitBox <gi...@apache.org>.
gmcrocetti edited a comment on issue #14758:
URL: https://github.com/apache/airflow/issues/14758#issuecomment-805248640


   
   Hey @ShawnMcGough . I'm worried about the lack of `Azure Data Factory` in this drop down menu. Do you see any exception being raised when you click in this menu ? The freshly inserted [hook_name](https://github.com/apache/airflow/blob/a7e144bec855f6ccf0fa5ae8447894195ffe170f/airflow/providers/microsoft/azure/hooks/azure_data_factory.py#L83) should've been there. @ephraimbuddy  @turbaszek, is there something I'm missing ? ProvidersManager seems straightforward with this return.
   
   Looks like there's an import error, at least in the latest version.  `Admin -> Connections # Conn Type *`
   ```shell
   Exception when importing 'airflow.providers.microsoft.azure.hooks.wasb.WasbHook' from 'apache-airflow-providers-microsoft-azure' package: No module named 'azure.storage.blob'
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] flvndh commented on issue #14758: Example DAG for New Azure Data Factory Hooks

Posted by GitBox <gi...@apache.org>.
flvndh commented on issue #14758:
URL: https://github.com/apache/airflow/issues/14758#issuecomment-798662442


   I'll see if I can make up one in the coming days 😉


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] gmcrocetti edited a comment on issue #14758: Example DAG for New Azure Data Factory Hooks

Posted by GitBox <gi...@apache.org>.
gmcrocetti edited a comment on issue #14758:
URL: https://github.com/apache/airflow/issues/14758#issuecomment-805248640


   Hey @ShawnMcGough . I'm worried about the lack of `Azure Data Factory` in this drop down menu. Do you see any exception being raised when you click in this menu ? The freshly inserted [hook_name](https://github.com/apache/airflow/blob/a7e144bec855f6ccf0fa5ae8447894195ffe170f/airflow/providers/microsoft/azure/hooks/azure_data_factory.py#L83) should've been there. @ephraimbuddy  @turbaszek, is there something I'm missing ? ProvidersManager seems straightforward with this return.
   
   Looks like there's an import error, at least in the latest Docker version.  `Admin -> Connections # Conn Type *`
   ```shell
   Exception when importing 'airflow.providers.microsoft.azure.hooks.wasb.WasbHook' from 'apache-airflow-providers-microsoft-azure' package: No module named 'azure.storage.blob'
   ```
   Other than that, version `2.0.1` is the one being installed by Docker, which doesn't contain DataFactory's hook. Shawn, did you run your test with a Docker environment ?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] josh-fell commented on issue #14758: Example DAG for New Azure Data Factory Hooks

Posted by GitBox <gi...@apache.org>.
josh-fell commented on issue #14758:
URL: https://github.com/apache/airflow/issues/14758#issuecomment-923340677


   This issue should be resolved now via #17885 -- perhaps partially at a minimum.  Along wIth the new operator and sensor for ADF, an example DAG was added to showcase these modules. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] gmcrocetti edited a comment on issue #14758: Example DAG for New Azure Data Factory Hooks

Posted by GitBox <gi...@apache.org>.
gmcrocetti edited a comment on issue #14758:
URL: https://github.com/apache/airflow/issues/14758#issuecomment-805248640


   Hey @ShawnMcGough . I'm worried about the lack of `Azure Data Factory` in this drop down menu. Do you see any exception being raised when you click in this menu ? The freshly inserted [hook_name](https://github.com/apache/airflow/blob/a7e144bec855f6ccf0fa5ae8447894195ffe170f/airflow/providers/microsoft/azure/hooks/azure_data_factory.py#L83) should've been there. @ephraimbuddy  @turbaszek, is there something I'm missing ? ProvidersManager seems straightforward with this return.
   
   Looks like there's an import error, at least in the latest Docker version.  `Admin -> Connections # Conn Type *`
   ```shell
   Exception when importing 'airflow.providers.microsoft.azure.hooks.wasb.WasbHook' from 'apache-airflow-providers-microsoft-azure' package: No module named 'azure.storage.blob'
   ```
   Other than that, version `2.0.1` is the one being installed by Docker, which doesn't contain DataFactory's hook. Did you run your test with a Docker environment, Shawn ?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] gmcrocetti edited a comment on issue #14758: Example DAG for New Azure Data Factory Hooks

Posted by GitBox <gi...@apache.org>.
gmcrocetti edited a comment on issue #14758:
URL: https://github.com/apache/airflow/issues/14758#issuecomment-805248640


   Hey @ShawnMcGough . I'm worried about the lack of `Azure Data Factory` in this drop down menu. Do you see any exception being raised when you click in this menu ? The freshly inserted [hook_name](https://github.com/apache/airflow/blob/a7e144bec855f6ccf0fa5ae8447894195ffe170f/airflow/providers/microsoft/azure/hooks/azure_data_factory.py#L83) should've been there. @ephraimbuddy  @turbaszek, is there something I'm missing ? ProvidersManager seems straightforward with this return.
   
   Looks like there's an import error, at least in the latest Docker version.  `Admin -> Connections # Conn Type *`
   ```shell
   Exception when importing 'airflow.providers.microsoft.azure.hooks.wasb.WasbHook' from 'apache-airflow-providers-microsoft-azure' package: No module named 'azure.storage.blob'
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ShawnMcGough commented on issue #14758: Example DAG for New Azure Data Factory Hooks

Posted by GitBox <gi...@apache.org>.
ShawnMcGough commented on issue #14758:
URL: https://github.com/apache/airflow/issues/14758#issuecomment-798808580


   I was able to cobble together an example below. What I struggled with was setting up the default connection. I wasn't able to find an `Azure Data Factory` connection type in the drop down. 
   ![image](https://user-images.githubusercontent.com/64036/111053565-cac3e880-8432-11eb-9aa4-3f8e41c61efc.png)
   
   I wasn't sure of the connection format, but found the following through viewing source code and trial & error:
   
   I selected `Azure` with the following settings.
   
   ```
   Conn Id=azure_data_factory_default
   Login=Client Id (from app registration)
   Password=Client Password (from app registration)
   Extra={"tenantId": "xxxx","subscriptionId": "xxxx"}
   ```
   Which worked fine.
   
   ```python
   from pprint import pprint
   from airflow.models import DAG
   from airflow.operators.python import PythonOperator
   from airflow.providers.microsoft.azure.hooks.azure_data_factory import AzureDataFactoryHook
   from airflow.utils.dates import days_ago
   
   
   def get_factory():
       """Get a data factory! (╯°□°)╯︵ ┻━┻"""
       hook = AzureDataFactoryHook()
       data_factory = hook.get_factory("resource-group", "factory-name")
       pprint(data_factory.name)
   
   
   with DAG("example_data_factory", schedule_interval="@once", start_date=days_ago(2)) as dag:
       create = PythonOperator(task_id="get-factory",
                               python_callable=get_factory)
   ```
   
   Happy to hear feedback on any improvements.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #14758: Example DAG for New Azure Data Factory Hooks

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #14758:
URL: https://github.com/apache/airflow/issues/14758#issuecomment-798502314


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] turbaszek commented on issue #14758: Example DAG for New Azure Data Factory Hooks

Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #14758:
URL: https://github.com/apache/airflow/issues/14758#issuecomment-798518067


   Maybe @gmcrocetti or @flvndh can help?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #14758: Example DAG for New Azure Data Factory Hooks

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #14758:
URL: https://github.com/apache/airflow/issues/14758#issuecomment-923340798


   Solved by https://github.com/apache/airflow/pull/17885


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ShawnMcGough commented on issue #14758: Example DAG for New Azure Data Factory Hooks

Posted by GitBox <gi...@apache.org>.
ShawnMcGough commented on issue #14758:
URL: https://github.com/apache/airflow/issues/14758#issuecomment-807351148


   @gmcrocetti 
   It looks like the issue I am having is that your [commit ](https://github.com/apache/airflow/commit/4372d456154a6922e0c0547a487af3cdadb43b4a) to address #14704 did not make it into the 1.2 release on pypi.org. 😞
   
   I pulled down the code from the 1.2 release on pypi.org and confirmed that the fix is not there.
   
   Is it possible to get a patch release for the fix?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] gmcrocetti edited a comment on issue #14758: Example DAG for New Azure Data Factory Hooks

Posted by GitBox <gi...@apache.org>.
gmcrocetti edited a comment on issue #14758:
URL: https://github.com/apache/airflow/issues/14758#issuecomment-805248640


   Hey @ShawnMcGough . I'm worried about the lack of `Azure Data Factory` in this drop down menu. Do you see any exception being raised when you click in this menu ? The freshly inserted [hook_name](https://github.com/apache/airflow/blob/a7e144bec855f6ccf0fa5ae8447894195ffe170f/airflow/providers/microsoft/azure/hooks/azure_data_factory.py#L83) should've been there. @ephraimbuddy  @turbaszek, is there something I'm missing ? ProvidersManager seems straightforward with this return.
   
   Looks like there's an import error, at least in the latest Docker version.  `Admin -> Connections # Conn Type *`
   ```shell
   Exception when importing 'airflow.providers.microsoft.azure.hooks.wasb.WasbHook' from 'apache-airflow-providers-microsoft-azure' package: No module named 'azure.storage.blob'
   ```
   Other than that, version `2.0.1` is the one being installed by Docker, which doesn't contain the DataFactory's hook. Did you run your test with a Docker environment, Shawn ?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org