You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/08 05:54:09 UTC
[GitHub] [airflow] BenGatewood opened a new issue #21412: v3.5.0 airflow.providers.microsoft.azure.operators.cosmos not running
BenGatewood opened a new issue #21412:
URL: https://github.com/apache/airflow/issues/21412
### Apache Airflow version
2.2.3 (latest released)
### What happened
Submitting this on advice from the community Slack: Attempting to use the v3.5.0 `AzureCosmosInsertDocumentOperator` operator fails with an attribute error: `AttributeError: 'CosmosClient' object has no attribute 'QueryDatabases'`
### What you expected to happen
Expected behaviour is that the document is upserted correctly. I've traced through the source and `does_database_exist()` seems to call `QueryDatabases()` on the result of `self.get_conn()`. Thing is `get_conn()` (AFAICT) returns an actual MS/AZ `CosmosClient` which definitely does not have a `QueryDatabases()` method (it's `query_databases()`)
### How to reproduce
From what I can see, any attempt to use this operator on airflow 2.2.3 will fail in this way
### Operating System
Ubuntu 18.04.5 LTS
### Versions of Apache Airflow Providers
azure-batch==12.0.0
azure-common==1.1.28
azure-core==1.22.0
azure-cosmos==4.2.0
azure-datalake-store==0.0.52
azure-identity==1.7.1
azure-keyvault==4.1.0
azure-keyvault-certificates==4.3.0
azure-keyvault-keys==4.4.0
azure-keyvault-secrets==4.3.0
azure-kusto-data==0.0.45
azure-mgmt-containerinstance==1.5.0
azure-mgmt-core==1.3.0
azure-mgmt-datafactory==1.1.0
azure-mgmt-datalake-nspkg==3.0.1
azure-mgmt-datalake-store==0.5.0
azure-mgmt-nspkg==3.0.2
azure-mgmt-resource==20.1.0
azure-nspkg==3.0.2
azure-storage-blob==12.8.1
azure-storage-common==2.1.0
azure-storage-file==2.1.0
msrestazure==0.6.4
### Deployment
Virtualenv installation
### Deployment details
Clean standalone install I am using for evaluating airflow for our environment
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on issue #21412: v3.5.0 airflow.providers.microsoft.azure.operators.cosmos not running
Posted by GitBox <gi...@apache.org>.
josh-fell commented on issue #21412:
URL: https://github.com/apache/airflow/issues/21412#issuecomment-1032785814
AFAICT This hook has been non-functional for a while. The `AzureCosmosDBHook` code looks to be aligned with [version 3.2 of the azure-cosmos package](https://pypi.org/project/azure-cosmos/3.2.0/) release in July 2020. However, [v4.0](https://pypi.org/project/azure-cosmos/4.0.0/) was already released in May 2020 which renamed these methods. I don't think this needs to be a blocker for RC2 since it's been broken for quite some time already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell edited a comment on issue #21412: v3.5.0 airflow.providers.microsoft.azure.operators.cosmos not running
Posted by GitBox <gi...@apache.org>.
josh-fell edited a comment on issue #21412:
URL: https://github.com/apache/airflow/issues/21412#issuecomment-1032732974
Oof. Looks like this is a widespread problem in the Azure Cosmos hook and needs to be cleaned up. I can log an issue for the clean up work in case folks want to help out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21412: v3.5.0 airflow.providers.microsoft.azure.operators.cosmos not running
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21412:
URL: https://github.com/apache/airflow/issues/21412#issuecomment-1032741935
Is it serious enough to skip Azure from release and how big of a fix it can be?
I can hold one or two more days with releasing RC2 for providers (to save time on coordination/release overhead) if this is something that we will likely have a fix in a day or two.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] BenGatewood commented on issue #21412: v3.5.0 airflow.providers.microsoft.azure.operators.cosmos not running
Posted by GitBox <gi...@apache.org>.
BenGatewood commented on issue #21412:
URL: https://github.com/apache/airflow/issues/21412#issuecomment-1033262724
OK I've got this working on a monkey-patched local version. Is there a style/form guide for PRs I should follow here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on issue #21412: v3.5.0 airflow.providers.microsoft.azure.operators.cosmos not running
Posted by GitBox <gi...@apache.org>.
josh-fell commented on issue #21412:
URL: https://github.com/apache/airflow/issues/21412#issuecomment-1033303480
There is a wonderfully [detailed doc on contributing](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst). It outlines the contribution workflow, PR guidelines, local testing options, `pre-commit` checks you should run before submitting a PR, etc. On Slack you can pose questions in the #airflow-how-to-pr channel as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #21412: v3.5.0 airflow.providers.microsoft.azure.operators.cosmos not running
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #21412:
URL: https://github.com/apache/airflow/issues/21412#issuecomment-1032394331
Assigned you @BenGatewood - fix is most welcome !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #21412: v3.5.0 airflow.providers.microsoft.azure.operators.cosmos not running
Posted by GitBox <gi...@apache.org>.
eladkal closed issue #21412:
URL: https://github.com/apache/airflow/issues/21412
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21412: v3.5.0 airflow.providers.microsoft.azure.operators.cosmos not running
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21412:
URL: https://github.com/apache/airflow/issues/21412#issuecomment-1032394331
Assigned yo @BenGatewood - fix is most welcome !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell edited a comment on issue #21412: v3.5.0 airflow.providers.microsoft.azure.operators.cosmos not running
Posted by GitBox <gi...@apache.org>.
josh-fell edited a comment on issue #21412:
URL: https://github.com/apache/airflow/issues/21412#issuecomment-1032785814
AFAICT This hook has been non-functional for a while. The `AzureCosmosDBHook` code looks to be aligned with [version 3.2 of the azure-cosmos package](https://pypi.org/project/azure-cosmos/3.2.0/) release in July 2020. However, [v4.0](https://pypi.org/project/azure-cosmos/4.0.0/) was already released in May 2020 which renamed these methods (very odd release sequence according to PyPI). I don't think this needs to be a blocker for RC2 since it's been broken for quite some time already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell edited a comment on issue #21412: v3.5.0 airflow.providers.microsoft.azure.operators.cosmos not running
Posted by GitBox <gi...@apache.org>.
josh-fell edited a comment on issue #21412:
URL: https://github.com/apache/airflow/issues/21412#issuecomment-1032785814
AFAICT This hook has been non-functional for a while. The `AzureCosmosDBHook` code looks to be aligned with [version 3.2 of the azure-cosmos package](https://pypi.org/project/azure-cosmos/3.2.0/) released in July 2020. However, [v4.0](https://pypi.org/project/azure-cosmos/4.0.0/) was already released in May 2020 which renamed these methods (very odd release sequence according to PyPI). I don't think this needs to be a blocker for RC2 since it's been broken for quite some time already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell edited a comment on issue #21412: v3.5.0 airflow.providers.microsoft.azure.operators.cosmos not running
Posted by GitBox <gi...@apache.org>.
josh-fell edited a comment on issue #21412:
URL: https://github.com/apache/airflow/issues/21412#issuecomment-1032732974
Oof. Looks like this is a widespread problem in the Azure Cosmos hook and needs to be cleaned up. I can log an issue for the clean up work in case folks wants to help out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21412: v3.5.0 airflow.providers.microsoft.azure.operators.cosmos not running
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21412:
URL: https://github.com/apache/airflow/issues/21412#issuecomment-1032925563
All right. Thanks for confirming @josh-fell ! . So no harm in releasing it now as it is not a regression. I proceed with RC2 then :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on issue #21412: v3.5.0 airflow.providers.microsoft.azure.operators.cosmos not running
Posted by GitBox <gi...@apache.org>.
josh-fell commented on issue #21412:
URL: https://github.com/apache/airflow/issues/21412#issuecomment-1032732974
Oof. Looks like this is a widespread problem in the Azure Cosmos hook and needs to be cleaned up. I can log an issue for the clean up work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #21412: v3.5.0 airflow.providers.microsoft.azure.operators.cosmos not running
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #21412:
URL: https://github.com/apache/airflow/issues/21412#issuecomment-1032239971
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org