You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/01/25 22:08:32 UTC
[GitHub] [airflow] tsaaii opened a new issue #21110: Connecting Airflow to Azure data factory fails giving a red banner but doesn't give the reason.
tsaaii opened a new issue #21110:
URL: https://github.com/apache/airflow/issues/21110
### Apache Airflow Provider(s)
microsoft-azure
### Versions of Apache Airflow Providers
apache-airflow-providers-microsoft-azure
### Apache Airflow version
2.2.3 (latest released)
### Operating System
windows
### Deployment
Other Docker-based deployment
### Deployment details
I have a docker image with kubernetes which has all the provider dependencies and other pip installations with official helm chart where I overwrote some values for airflow which are mostly credentials related.
### What happened
I am trying to test my azure data factory connection and when I hit the test button after giving client ID, secret, Factory name, resource group name, subscription ID , it displays a red banner but it doesn't say what failed.
I am looking to debug this.
My azure connector configuration
red banner indicating something isn't right but it doesn't say what
### What you expected to happen
The goal is to see a successful connection to ADF from Airflow so that i can run my ADF pipelines on airflow.
### How to reproduce
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell edited a comment on issue #21110: Connecting Airflow to Azure data factory fails giving a red banner but doesn't give the reason.
Posted by GitBox <gi...@apache.org>.
josh-fell edited a comment on issue #21110:
URL: https://github.com/apache/airflow/issues/21110#issuecomment-1021776570
I can't necessarily speak to why there is an empty red banner but I can see two things going on:
1. The "Test" button is inactive so you actually can't test the connection. Typically you should see "UNAUTHORIZED" in the red banner and the tooltip below when you hover over the "Test" button:
2. The Azure Data Factory connection does not have a `test_connection()` method implemented so even if you did enable the Airflow REST API, you still wouldn't be able to test the connection. Here is documentation on [testing connections from the UI](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#test-connections).
FYI there has been an enhancement to also dynamically deactivate the "Test" button if the associated hook/connection type does not have a `test_connection()` method available via PR #19792 but it has not been released yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #21110: Connecting Airflow to Azure data factory fails giving a red banner but doesn't give the reason.
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #21110:
URL: https://github.com/apache/airflow/issues/21110#issuecomment-1021655286
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell edited a comment on issue #21110: Connecting Airflow to Azure data factory fails giving a red banner but doesn't give the reason.
Posted by GitBox <gi...@apache.org>.
josh-fell edited a comment on issue #21110:
URL: https://github.com/apache/airflow/issues/21110#issuecomment-1021776570
I can't necessarily speak to why there is an empty red banner but I can see two things going on:
1. The "Test" button is inactive so you actually can't test the connection. Typically you should see "UNAUTHORIZED" in the red banner when clicking on the inactive button and the tooltip below when you hover over the "Test" button:
2. The Azure Data Factory connection does not have a `test_connection()` method implemented so even if you did enable the Airflow REST API, you still wouldn't be able to test the connection. Here is [documentation on testing connections](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#test-connections).
FYI there has been an enhancement to also dynamically deactivate the "Test" button if the associated hook/connection type does not have a `test_connection()` method available via PR #19792 but it has not been released yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] wesleyemery commented on issue #21110: Connecting Airflow to Azure data factory fails giving a red banner but doesn't give the reason.
Posted by GitBox <gi...@apache.org>.
wesleyemery commented on issue #21110:
URL: https://github.com/apache/airflow/issues/21110#issuecomment-1024505721
I've was able to enable the test button by enabling the airflow API in the helm chart:
```
# airflowVersion: "2.2.2"
# defaultAirflowTag: "2.2.2"
executor: "KubernetesExecutor"
images:
airflow:
repository: "beertechcontainerregistrydev.azurecr.io/beertech-airflow-dev"
tag: "dev-6b0380de7b8c4ee2b2f73d64e25895d99342fe53"
useDefaultImageForMigration: true
nodeSelector:
agentpool: "airflow"
ingress:
enabled: true
web:
annotations:
kubernetes.io/ingress.class: "nginx"
hosts: ["airflow.dev.beertech.com"]
tls:
enabled: true
secretName: ingress-tls-secret
# flower:
# annotations:
# kubernetes.io/ingress.class: "nginx"
# path: "/flower"
# hosts: ["dev.airflow.beertech.com"]
# tls:
# enabled: true
# secretName: ingress-tls-secret
config:
api:
auth_backend: airflow.api.auth.backend.default
webserver:
defaultUser:
enabled: true
role: Admin
username: devops
email:
firstName: devops
lastName: admin
password:
dags:
gitSync:
enabled: true
repo: "ssh://git@github.com/ab-inbev-beertech/beertech-airflow.git"
branch: "main"
subPath: "dags/"
sshKeySecret: airflow-ssh-secret
extraSecrets:
airflow-ssh-secret:
data: |
```
However, when I test the connection I still get a red banner with no info for why it's failing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] wesleyemery edited a comment on issue #21110: Connecting Airflow to Azure data factory fails giving a red banner but doesn't give the reason.
Posted by GitBox <gi...@apache.org>.
wesleyemery edited a comment on issue #21110:
URL: https://github.com/apache/airflow/issues/21110#issuecomment-1024505721
I've was able to enable the test button by enabling the airflow API in the helm chart:
```
# airflowVersion: "2.2.2"
# defaultAirflowTag: "2.2.2"
executor: "KubernetesExecutor"
images:
airflow:
repository: ""
tag: ""
useDefaultImageForMigration: true
nodeSelector:
agentpool: "airflow"
ingress:
enabled: true
web:
annotations:
kubernetes.io/ingress.class: "nginx"
hosts: ["airflow.dev.beertech.com"]
tls:
enabled: true
secretName: ingress-tls-secret
# flower:
# annotations:
# kubernetes.io/ingress.class: "nginx"
# path: "/flower"
# hosts: ["dev.airflow.beertech.com"]
# tls:
# enabled: true
# secretName: ingress-tls-secret
config:
api:
auth_backend: airflow.api.auth.backend.default
webserver:
defaultUser:
enabled: true
role: Admin
username: devops
email:
firstName: devops
lastName: admin
password:
dags:
gitSync:
enabled: true
repo: ""
branch: "main"
subPath: "dags/"
sshKeySecret: airflow-ssh-secret
extraSecrets:
airflow-ssh-secret:
data: |
```
However, when I test the connection I still get a red banner with no info for why it's failing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on issue #21110: Connecting Airflow to Azure data factory fails giving a red banner but doesn't give the reason.
Posted by GitBox <gi...@apache.org>.
josh-fell commented on issue #21110:
URL: https://github.com/apache/airflow/issues/21110#issuecomment-1021776570
I can't _necessarily_ speak to why there is an empty red banner but I can see two things going on:
1. The "Test" button is inactive so you actually can't test the connection. Typically you should see "UNAUTHORIZED" in the red banner and the tooltip below when you hover over the "Test" button:
2. The Azure Data Factory connection does not have a `test_connection()` method implemented so even if you did enable the Airflow REST API, you still wouldn't be able to test the connection. Here is documentation on [testing connections from the UI](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#test-connections).
FYI there has been an enhancement to also dynamically deactivate the "Test" button if the associated hook/connection type does not have a `test_connection()` method available via PR #19792 but it has not been released yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell edited a comment on issue #21110: Connecting Airflow to Azure data factory fails giving a red banner but doesn't give the reason.
Posted by GitBox <gi...@apache.org>.
josh-fell edited a comment on issue #21110:
URL: https://github.com/apache/airflow/issues/21110#issuecomment-1021776570
I can't necessarily speak to why there is an empty red banner but I can see two things going on:
1. The "Test" button is inactive so you actually can't test the connection. Typically you should see "UNAUTHORIZED" in the red banner when clicking on the inactive button and the tooltip below when you hover over the "Test" button:
2. The Azure Data Factory connection does not have a `test_connection()` method implemented so even if you did enable the Airflow REST API, you still wouldn't be able to test the connection. Here is [documentation on testing connections](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#test-connections).
FYI there is an enhancement to also dynamically deactivate the "Test" button if the associated hook/connection type does not have a `test_connection()` method available via PR #19792 but it has not been released yet. This enhancement also functionally disables the button instead of only the appearance of the button being disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell edited a comment on issue #21110: Connecting Airflow to Azure data factory fails giving a red banner but doesn't give the reason.
Posted by GitBox <gi...@apache.org>.
josh-fell edited a comment on issue #21110:
URL: https://github.com/apache/airflow/issues/21110#issuecomment-1026437147
You can certainly create a `PythonOperator`/TaskFlow function that calls one of the `AzureDataFactoryHook` methods (e.g. `get_factory()`, or something else straightforward) and see if you can retrieve pertinent data. But there is really no way to test _prior to_ executing an Airflow task like you can in the UI if it was all configured for ADF.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on issue #21110: Connecting Airflow to Azure data factory fails giving a red banner but doesn't give the reason.
Posted by GitBox <gi...@apache.org>.
josh-fell commented on issue #21110:
URL: https://github.com/apache/airflow/issues/21110#issuecomment-1026437147
You can certainly create a `PythonOperator`/TaskFlow function that calls one of the `AzureDataFactoryHook` methods (e.g. `get_factory()`, or something else straightforward) and see if you can retrieve pertinent data. But there is really no way to test _prior to_ executing an Airflow task like you can in the UI for ADF.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] wesleyemery commented on issue #21110: Connecting Airflow to Azure data factory fails giving a red banner but doesn't give the reason.
Posted by GitBox <gi...@apache.org>.
wesleyemery commented on issue #21110:
URL: https://github.com/apache/airflow/issues/21110#issuecomment-1024582826
Is there any other way to test the connection? Would DAG logging work [Stack overflow DAG logging](https://stackoverflow.com/questions/40120467/adding-logs-to-airflow-logs)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell edited a comment on issue #21110: Connecting Airflow to Azure data factory fails giving a red banner but doesn't give the reason.
Posted by GitBox <gi...@apache.org>.
josh-fell edited a comment on issue #21110:
URL: https://github.com/apache/airflow/issues/21110#issuecomment-1021776570
I can't necessarily speak to why there is an empty red banner but I can see two things going on:
1. The "Test" button is inactive so you actually can't test the connection. Typically you should see "UNAUTHORIZED" in the red banner and the tooltip below when you hover over the "Test" button:
2. The Azure Data Factory connection does not have a `test_connection()` method implemented so even if you did enable the Airflow REST API, you still wouldn't be able to test the connection. Here is [documentation on testing connections](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#test-connections).
FYI there has been an enhancement to also dynamically deactivate the "Test" button if the associated hook/connection type does not have a `test_connection()` method available via PR #19792 but it has not been released yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell edited a comment on issue #21110: Connecting Airflow to Azure data factory fails giving a red banner but doesn't give the reason.
Posted by GitBox <gi...@apache.org>.
josh-fell edited a comment on issue #21110:
URL: https://github.com/apache/airflow/issues/21110#issuecomment-1021776570
I can't necessarily speak to why there is an empty red banner but I can see two things going on:
1. The "Test" button is inactive so you actually can't test the connection. Typically you should see "UNAUTHORIZED" in the red banner when clicking on the inactive button and the tooltip below when you hover over the "Test" button:
2. The Azure Data Factory connection does not have a `test_connection()` method implemented so even if you did enable the Airflow REST API, you still wouldn't be able to test the connection. Here is [documentation on testing connections](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#test-connections).
FYI there is an enhancement to also dynamically deactivate the "Test" button if the associated hook/connection type does not have a `test_connection()` method available via PR #19792 but it has not been released yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on issue #21110: Connecting Airflow to Azure data factory fails giving a red banner but doesn't give the reason.
Posted by GitBox <gi...@apache.org>.
josh-fell commented on issue #21110:
URL: https://github.com/apache/airflow/issues/21110#issuecomment-1024531680
Right on. Even if you enabled the REST API, the second point still holds. The underlying `AzureDataFactoryHook` doesn't implement a `test_connection()` method which the API uses to perform the test. Without that method, you can't test the connection.
Unfortunately, I tried to whip something up for testing the ADF connection and ran into #21188 which needs to be fixed first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org