You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/07 10:13:32 UTC
[GitHub] [airflow] dinowernli opened a new issue #21377: Databricks: add endpoint CRUD operators
dinowernli opened a new issue #21377:
URL: https://github.com/apache/airflow/issues/21377
### Description
Operators to manipulate Databricks SQL endpoints.
### Use case/motivation
Support conveniently managing the lifecycle of Databricks SQL endpoints. This likely entails building one or more operators which talk to the [endpoints API](https://docs.databricks.com/sql/api/sql-endpoints.html) and support:
* Creating endpoints
* Deleting endpoints
* Starting endpoints
* Stopping endpoints
These operations map cleanly to individual API calls.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on issue #21377: Databricks: add SQL endpoint operators
Posted by GitBox <gi...@apache.org>.
alexott commented on issue #21377:
URL: https://github.com/apache/airflow/issues/21377#issuecomment-1053696353
@besenthil is implementation in progress, or you need an additional help with it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] besenthil commented on issue #21377: Databricks: add SQL endpoint operators
Posted by GitBox <gi...@apache.org>.
besenthil commented on issue #21377:
URL: https://github.com/apache/airflow/issues/21377#issuecomment-1037626693
@dinowernli
Is there a test stub that I can use for testing the API call?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on issue #21377: Databricks: add SQL endpoint operators
Posted by GitBox <gi...@apache.org>.
alexott commented on issue #21377:
URL: https://github.com/apache/airflow/issues/21377#issuecomment-1038881564
@besenthil we don't have stubs for testing, so you need to have access to a workspace with Dataricks SQL, plus corresponding permissions to create SQL Endpoints.
From implementation perspective, you need to use [`_do_api_call` function](https://github.com/apache/airflow/blob/main/airflow/providers/databricks/hooks/databricks.py#L294) - first POST definition of SQL Endpoint, and then check for it's status. Please note that request will fail if you submit an endpoint with the same name, so you will need to check it (I have a code for that in PR for Databricks SQL Operator)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] besenthil commented on issue #21377: Databricks: add SQL endpoint operators
Posted by GitBox <gi...@apache.org>.
besenthil commented on issue #21377:
URL: https://github.com/apache/airflow/issues/21377#issuecomment-1037626693
@dinowernli
Is there a test stub that I can use for testing the API call?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dinowernli commented on issue #21377: Databricks: add SQL endpoint operators
Posted by GitBox <gi...@apache.org>.
dinowernli commented on issue #21377:
URL: https://github.com/apache/airflow/issues/21377#issuecomment-1032530028
Right now, the logic in that function is specific to the kind of operators defined in the same file, i.e., to operators which create runs. I think something similar would be useful for other operators, but it would likely have to operate on the status of a query execution or similar (rather than the state of a job run).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] besenthil commented on issue #21377: Databricks: add SQL endpoint operators
Posted by GitBox <gi...@apache.org>.
besenthil commented on issue #21377:
URL: https://github.com/apache/airflow/issues/21377#issuecomment-1032500586
@dinowernli
Would the function `_handle_databricks_operator_execution` be applicable for API endpoints which are not ASYNC like the creation of sql endpoint?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] besenthil commented on issue #21377: Databricks: add SQL endpoint operators
Posted by GitBox <gi...@apache.org>.
besenthil commented on issue #21377:
URL: https://github.com/apache/airflow/issues/21377#issuecomment-1045915856
Thanks @alexott
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] besenthil commented on issue #21377: Databricks: add SQL endpoint operators
Posted by GitBox <gi...@apache.org>.
besenthil commented on issue #21377:
URL: https://github.com/apache/airflow/issues/21377#issuecomment-1053742811
Hi Alex,
Yes it is in progress.
On Mon, 28 Feb, 2022, 3:29 am Alex Ott, ***@***.***> wrote:
> @besenthil <https://github.com/besenthil> is implementation in progress,
> or you need an additional help with it?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/airflow/issues/21377#issuecomment-1053696353>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAHSRRYDFG7INIMAJBE24VDU5KNFRANCNFSM5NW7VFOA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] besenthil commented on issue #21377: Databricks: add SQL endpoint operators
Posted by GitBox <gi...@apache.org>.
besenthil commented on issue #21377:
URL: https://github.com/apache/airflow/issues/21377#issuecomment-1032754148
Thanks @dinowernli I will take a look at it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org